title: ImportError: gpu_nmscpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration

body:
I am trying code from this repo.
I am facing the following error when I run demo_video_mobile.py
`Traceback (most recent call last):
  File "demo_video_mobile.py", line 28, in <module>
    from nms.gpu_nms import gpu_nms
ImportError: /home/peeterson/Desktop/lighttrack/HPE/../lib/nms/gpu_nms.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration
`

module versions inside py36 environment:
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
_tflow_180_select         1.0                         gpu    anaconda
absl-py                   0.2.2                    py36_0    anaconda
astor                     0.6.2                    py36_0    anaconda
blas                      1.0                         mkl    conda-forge
bleach                    1.5.0                    py36_0    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2020.12.5            ha878542_0    conda-forge
cairo                     1.14.12              h77bcde2_0    anaconda
certifi                   2020.12.5        py36h5fab9bb_1    conda-forge
cloudpickle               0.5.3                    pypi_0    pypi
cmake                     3.12.0               h307fef2_0    anaconda
cudatoolkit               9.0                  h13b8566_0    anaconda
cudnn                     7.1.2                 cuda9.0_0    anaconda
cupti                     9.0.176                       0    anaconda
cycler                    0.10.0                   py36_0    conda-forge
cython                    0.28.3                   pypi_0    pypi
dask                      0.17.5                   pypi_0    pypi
dataclasses               0.8                      pypi_0    pypi
datetime                  4.2                      pypi_0    pypi
dbus                      1.13.0               h3a4f0e9_0    conda-forge
decorator                 4.3.0                    pypi_0    pypi
easydict                  1.7                      pypi_0    pypi
expat                     2.2.5                         0    conda-forge
ffmpeg                    3.2.4                         3    conda-forge
fontconfig                2.13.0                        0    conda-forge
freetype                  2.8.1                         0    conda-forge
gast                      0.2.0                    py36_0    anaconda
giflib                    5.1.4                         0    conda-forge
glib                      2.53.6               h5d9569c_2    anaconda
graphite2                 1.3.11                        0    conda-forge
grpcio                    1.12.0           py36hdbcaa40_0    anaconda
gst-plugins-base          1.12.4               h33fb286_0    anaconda
gstreamer                 1.12.4               hb53b477_0    anaconda
h5py                      2.9.0                    pypi_0    pypi
harfbuzz                  1.7.6                hc5b324e_0    anaconda
hdf5                      1.10.2                        0    conda-forge
html5lib                  0.9999999                py36_0    conda-forge
icu                       58.2                          0    conda-forge
imageio                   2.3.0                    py36_0    conda-forge
jasper                    1.900.1                       4    conda-forge
jpeg                      9b                   h024ee3a_2
kiwisolver                1.0.1                    py36_1    conda-forge
libcurl                   7.61.0               h1ad7b7a_0    anaconda
libedit                   3.1.20170329         h6b74fdf_2    anaconda
libffi                    3.2.1                hd88cf55_4    anaconda
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    anaconda
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    anaconda
libiconv                  1.15                 h63c8f33_5    anaconda
libopenblas               0.2.20               h9ac9557_7    anaconda
libpng                    1.6.34                        0    conda-forge
libprotobuf               3.5.2                h6f1eeef_0    anaconda
libssh2                   1.8.0                h9cfc8f7_4    anaconda
libstdcxx-ng              7.2.0                hdf63c60_3    anaconda
libtiff                   4.0.9                he6b73bb_1    conda-forge
libuuid                   1.0.3                h1bed415_2    anaconda
libwebp                   0.5.2                         7    conda-forge
libxcb                    1.13                          0    conda-forge
libxml2                   2.9.8                h26e45fe_1    anaconda
llvm-openmp               11.0.1               h4bd325d_0    conda-forge
markdown                  2.6.11                   py36_0    anaconda
matplotlib                2.2.2            py36h0e671d2_0    anaconda
mkl                       2020.4             h726a3e6_304    conda-forge
msgpack                   0.5.6                    pypi_0    pypi
msgpack-numpy             0.4.3                    pypi_0    pypi
munkres                   1.0.12                   pypi_0    pypi
ncurses                   6.1                  hf484d3e_0    anaconda
networkx                  2.1                      pypi_0    pypi
numpy                     1.14.3           py36h28100ab_1    anaconda
numpy-base                1.14.3           py36h0ea5e3f_1    anaconda
olefile                   0.45.1                   py36_0    conda-forge
openblas                  0.2.20                        8    conda-forge
opencv                    3.4.1            py36h6fd60c2_1    anaconda
opencv-python             3.4.1.15                 pypi_0    pypi
openssl                   1.0.2o                        0    conda-forge
pcre                      8.41                          1    conda-forge
pillow                    5.1.0                    py36_0    conda-forge
pip                       21.0.1                   pypi_0    pypi
pixman                    0.34.0                        2    conda-forge
protobuf                  3.5.2            py36hf484d3e_0    anaconda
pyarrow                   0.9.0                    pypi_0    pypi
pycocotools               2.0.0                    pypi_0    pypi
pyparsing                 2.2.0                    py36_0    conda-forge
pyqt                      5.6.0                    py36_5    conda-forge
pyqt5                     5.12.1                   pypi_0    pypi
pyqt5-sip                 4.19.15                  pypi_0    pypi
python                    3.6.5                hc3d631a_2    anaconda
python-dateutil           2.7.3                      py_0    conda-forge
python_abi                3.6                     1_cp36m    conda-forge
pytz                      2018.4                     py_0    conda-forge
pywavelets                0.5.2                    pypi_0    pypi
pyyaml                    3.12                     pypi_0    pypi
pyzmq                     17.0.0                   pypi_0    pypi
qt                        5.6.2               hd25b39d_14    anaconda
readline                  7.0                  ha6073c6_4    anaconda
rhash                     1.3.6                hb7f436b_0    anaconda
scikit-image              0.14.0                   pypi_0    pypi
scipy                     1.1.0                    pypi_0    pypi
setproctitle              1.1.10                   pypi_0    pypi
setuptools                39.2.0                   py36_0    conda-forge
shapely                   1.6.4.post1              pypi_0    pypi
sip                       4.18                     py36_1    conda-forge
six                       1.11.0           py36h372c433_1    anaconda
sqlite                    3.23.1               he433501_0    anaconda
tabulate                  0.8.2                    pypi_0    pypi
tensorboard               1.8.0            py36hf484d3e_0    anaconda
tensorflow                1.8.0                hb11d968_0    anaconda
tensorflow-base           1.8.0            py36hc1a7637_0    anaconda
tensorflow-gpu            1.8.0                h7b35bdc_0    anaconda
termcolor                 1.1.0                    py36_1    conda-forge
tk                        8.6.7                hc745277_3    anaconda
toolz                     0.9.0                    pypi_0    pypi
torch                     1.0.1                    pypi_0    pypi
torchvision               0.2.2                    pypi_0    pypi
tornado                   5.0.2                    py36_0    conda-forge
tqdm                      4.19.9                   pypi_0    pypi
typing-extensions         3.7.4.3                  pypi_0    pypi
webencodings              0.5.1                      py_1    conda-forge
werkzeug                  0.14.1                   py36_0    anaconda
wheel                     0.31.1                   py36_0    conda-forge
x264                      20131218                      0    conda-forge
xorg-libxau               1.0.8                         3    conda-forge
xorg-libxdmcp             1.1.2                         3    conda-forge
xz                        5.2.4                h14c3975_4    anaconda
zlib                      1.2.11               ha838bed_2    anaconda
zope-interface            4.5.0                    pypi_0    pypi

I have installed the same cuda and cudnn versions above in my system.
I have exported the path as well.
export PATH=/usr/local/cuda-9.0/bin export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64:/usr/local/cuda-9.0/extras/CUPTI/lib64:/lib/nccl/cuda-9
I also tried to use an earlier version of gcc as per this:
gcc cuda compatibility
when i run
 nvcc --version

    nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

I still get the same error. could someone suggest how to fix this?


title: How to train Spacy3 project with FP16 mixed precision

body:
The goal is to run python -m spacy train with FP16 mixed precision to enable the use of large transformers (roberta-large, albert-large, etc.) in limited VRAM (RTX 2080ti 11 GB).
The new Spacy3 project.yml approach to training directly uses Huggingface-transformers models loaded via Spacy-transformers v1.0. Huggingface models can be run with mixed precision just by adding the --fp16 flag (as described here).
The spacy config was generated using python -m spacy init config --lang en --pipeline ner --optimize efficiency --gpu -F default.cfg, and checked to be complete by python -m spacy init fill-config default.cfg config.cfg --diff. Yet no FP16 / mixed-precision is to be found.
To reproduce
Use the spaCy Project: Named Entity Recognition (WikiNER) with changed init-config in project.yml to use a GPU and a transformer (roberta-base by default):
commands:
  -
    name: init-config
    help: "Generate a transformer English NER config"
    script:
      - "python -m spacy init config --lang en --pipeline ner --gpu -F --optimize efficiency -C configs/${vars.config}.cfg"

What was tested

Added --fp16 to python -m spacy project run
Added --fp16 to python -m spacy train
Added fp16 = true to default.cfg in various sections ([components.transformer], [components.transformer.model], [training], [initialize])

The logic was transformers are run in FP16 as such:
from transformers import TrainingArguments
TrainingArguments(..., fp16=True, ...)

SW stack specifics
 - spacy              3.0.3
 - spacy-transformers 1.0.1
 - transformers       4.2.2
 - torch              1.6.0+cu101


title: Why does Pytorch EmbeddingBag with mode "max" not accept `per_sample_weights`?

body:
Pytorch's EmbeddingBag allows for efficient lookup + reduce operations on varying length collections of embedding indices. There are 3 modes: "sum", "average" and "max" for the reduce operation. With "sum", you can also provide per_sample_weights giving you a weighted sum.
Why is per_sample_weights not allowed for the "max" operation? Looking at how it's implemented, I can only assume there is an issue with performing a "ReduceMean" or "ReduceMax" operation after a "Mul" operation. Could that be something to do with calculating gradients??

p.s: It's easy enough to turn a weighted sum into a weighted average by dividing by the sum of the weights, but for "max" you can't get a weighted equivalent like that.


title: ValueError: Can't optimize a non-leaf Tensor,but requires_grad is true

body:
I am trying to optimize self.parameters() with torch.optim.adam.Here is the original code:
    def learning_rate_adjust(self, args):
      self.learning_rate *= 0.8
      self.train_op = torch.optim.Adam(self.parameters(), lr=self.learning_rate, weight_decay=args.l2_reg)

but when I run this code,here is the bug:
Traceback (most recent call last):
   File "main.py", line 37, in <module>
      train(args, loader, train_set_length, test_seq, item_attr_set, [], [])
   File "E:\ACAM\0324ACAM-model-master\ACAM-model\code\train.py", line 62, in train
      net.learning_rate_adjust(args)
   File "E:\ACAM\0324ACAM-model-master\ACAM-model\code\model.py", line 160, in learning_rate_adjust
      self.train_op = torch.optim.Adam(self.parameters(), lr=self.learning_rate, weight_decay=args.l2_reg)
   File "D:\Anaconda3\envs\bio\lib\site-packages\torch\optim\adam.py", line 42, in __init__
      super(Adam, self).__init__(params, defaults)
   File "D:\Anaconda3\envs\bio\lib\site-packages\torch\optim\optimizer.py", line 51, in __init__
      self.add_param_group(param_group)
   File "D:\Anaconda3\envs\bio\lib\site-packages\torch\optim\optimizer.py", line 202, in
      add_param_group
      raise ValueError("can't optimize a non-leaf Tensor")
ValueError: can't optimize a non-leaf Tensor

so I try to see the list of self.parameters(),and i found list(self.parameters())[9] is a a non-leaf Tensor,then like some similar others' questions,i print the code like:
print('9',list(self.parameters())[9].requires_grad)
>>>true
print('9',list(self.parameters())[9].is_leaf)
>>>false

so i dont know how to change my code,please help me.T T


title: Why use Variable() in inference?

body:
I am learning PyTorch for an image classification task, and I ran into code where someone used a PyTorch Variable() in their function for prediction:
def predict_image(image):
    image_tensor = test_transforms(image).float()
    image_tensor = image_tensor.unsqueeze_(0)
    input = Variable(image_tensor)
    input = input.to(device)
    output = model(input)
    index = output.data.cpu().numpy().argmax()
    return index

Why do they use Variable() here? (even though it works fine without it.)


I am trying to run my PyTorch code on a Ubuntu server, it works well on my own computer, but it failed to run on the server.
Is this because of something related to PyTorch version?
This problem seems typical but yet no solutions work.
Traceback (most recent call last):
  File "train.py", line 12, in <module>
    from data_manager import *
  File "/data1/lijun/cross_modal_reid_bigma/transformer/data_manager.py", line 7, in <module>
    from util.data_loader import DataLoader
  File "/data1/lijun/cross_modal_reid_bigma/transformer/util/data_loader.py", line 6, in <module>
    from torchtext.legacy.data import Field, BucketIterator
  File "/usr/local/anaconda3/lib/python3.6/site-packages/torchtext/__init__.py", line 3, in <module>
    from . import datasets
  File "/usr/local/anaconda3/lib/python3.6/site-packages/torchtext/datasets/__init__.py", line 2, in <module>
    from .ag_news import AG_NEWS
  File "/usr/local/anaconda3/lib/python3.6/site-packages/torchtext/datasets/ag_news.py", line 2, in <module>
    from torchtext.data.datasets_utils import _RawTextIterableDataset
  File "/usr/local/anaconda3/lib/python3.6/site-packages/torchtext/data/datasets_utils.py", line 205, in <module>
    class _RawTextIterableDataset(torch.utils.data.IterableDataset):
AttributeError: module 'torch.utils' has no attribute 'data'


title: Lstm for continuos video

body:
Can we input a continuous video which contains sequences of both positive classes and negative classes to train LSTM, on several thousand of such videos?

  My overall objective is to mark videos realtime with particular
  scenes(e.g. if I’ve 0-100 frames and frame number 30-60 contains some
  yoga scenes, I need to mark them)

Right now the approach which I’m following is to split the video into two categories of positive sequences and negative sequences and train LSTM (on top of Mobnet CNN, FC replaced by LSTM layers).

  But somehow this does not give any improvement compared to Mobnet
  alone when we run evaluation on non-split videos.

Both Mobnet and LSTM are trained separately. I save output of Mobnet(FC removed) in numpy arrays and then read these arrays for training LSTM.

Here is the sample of code used for this approach:

epochs = 250
batch_size = 128

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        in_size = 1024
        classes_no = 2
        hidden_size = 512
        layer_no = 2

        self.lstm = nn.LSTM(in_size, hidden_size, layer_no, batch_first=True)
        self.linear = nn.Linear(hidden_size, classes_no)

    def forward(self, input_seq):
        output_seq, _ = self.lstm(input_seq)
        last_output = output_seq[:,-1]
        class_predictions = self.linear(last_output)
        return class_predictions

def nploader(npfile):
    a = np.load(npfile)
    return a

def train():
    npdataloader = torchvision.datasets.DatasetFolder('./featrs/',
            nploader, ['npy'], transform=None, target_transform=None)

    data_loader = torch.utils.data.DataLoader(npdataloader,
                                              batch_size=batch_size,
                                              shuffle=False,
                                              num_workers=1)

    model = Model().cuda()
    loss = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr = 0.001)
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.8)
    model.train()

    for epoch in range(0, epochs):
        for input_seq, target in data_loader:
            optimizer.zero_grad()
            output = model(input_seq.cuda())
            err = loss(output.cuda(), target.cuda())
            err.backward()
            optimizer.step()
        scheduler.step()

    torch.save(model.state_dict(), 'lstm.ckpt')


title: How to use multiple cores in Caffe2 on mobile?

body:
I’ve got a simple model consisting only of convolutions (even no activation between) and I wanted to benchmark it in Caffe2 on ARM Android device using multiple cores.

When I run

./speed_benchmark --init_net=model_for_inference-simplified-init-net.pb --net=model_for_inference-simplified-predict-net.pb --iter=1

it runs on single core.

Speed benchmark was built using:

scripts/build_android.sh -DANDROID_ABI=arm64-v8a -DANDROID_TOOLCHAIN=clang -DBUILD_BINARY=ON

On X86 it has been build via

mkdir build
cd build
cmake .. -DBUILD_BINARY=ON

and setting OMP_NUM_THREADS=8 helps but not on ARM

Do I need to change the building command for arm, set some environmental variables, some binary arguments or something else?


title: When to use learning rate finder

body:
Reading the paper '
Cyclical Learning Rates for Training Neural Networks' https://arxiv.org/abs/1506.01186

Does it make sense to use the learning rate finder if the model is over-fitting ? Other than reduce the number of iterations before the model overfit's will using the learning finder prevent over-fitting ?

From reading the paper there is no suggestion this method of reduces over-fitting, is my interpretation correct ?


title: How can I make a non-layered feed forward NN in pytorch?

body:
I want to set up a network that may have lots of complicated skip connections, but will still be feed forward (no recurrent connections). Each node will still be a "classic" node in the sense that it will just have a weight for each connection, and then multiply each weight by the corresponding input, sum them, and put it through a nonlinear function.

For example, it might look like this:

However, I don't think I can easily use something like the nn.Linear module to calculate outputs, because (in the picture for example), the red nodes get inputs from the green nodes and the pink nodes, but the green nodes also need the pink nodes' outputs to calculate.

I can think of some ways to do this, but they're real ugly and pytorch usually has a smart way of doing things. Is there a good way to do this?


title: Mapping predictions of partially overlapping batches

body:
I'm working with pytorch_pretrained_bert for NER based on the implementation available here: https://github.com/kamalkraj/BERT-NER
When predicting on longer texts, the predict function in bert.py in there returns an out of index error. My current understanding is that this is due to the nr of bert tokens (which are not necessarily the same, and indeed in many cases are not the same as the number of tokens for ex. nltk.word_tokenize() returns) makes out of the input text. If this exceeds 512 (as this is the default length for the pre-trained models, if I understood correctly), it returns this error (and works fine for shorter input).
So, I decided to implement a wrapper that, in case I have more bert tokens (i.e. the stuff bert's own tokenizer returns) than 512, generates batches that partially overlap. The overlap is desired, because just cutting it up into pieces, padding the last one (as it is unlikely that my nr of tokens is divisible by 512, hence the last, shorter batch would require padding) loses context. It means, however, that I have to do predictions batch-wise, and then put things back together afterwards.
This cutting up into batches (of length 512) is straightforward enough. I'm currently working on the putting back together predictions part.
For this, I wrote the following code:
def map_batches(batches, blen):
    master = []
    for i in range(len(batches)-1):
        appendbatch = []
        # if first round, append first bit of current batch with only one prediction
        if i == 0:
            appendbatch = batches[0][:int(blen/2)-1]
        thisbatch = batches[i]
        nextbatch = batches[i+1]
        offset = int(blen/2)-1 # think -1 is needed for zero offsetting, but not sure, test/debug!
        for j in range(offset, blen-offset+2): # and debug +2 here too!
            thistuple = thisbatch[j]
            nexttuple = nextbatch[j-offset]
            if thistuple[1] > nexttuple[1]:
                thisbatch[j] = thistuple
            else:
                thisbatch[j] = nexttuple
        appendbatch.extend(thisbatch[int(blen/2)-1:])
        # and at end add last bit of last batch with only one prediction
        if i == len(batches)-2:
            appendbatch.extend(nextbatch[int(blen/2)+1:])
        master.append(appendbatch)
    return master

blen = 6
a = [('I-LOC', 0.1),
     ('O', 0.2),
     ('I-LOC', 0.5),
     ('O', 0.2),
     ('I-LOC', 0.9),
     ('O', 0.1)]

b = [('B-LOC', 0.6),
     ('O', 0.7),
     ('I-LOC', 0.8),
     ('O', 0.9),
     ('O', 0.99),
     ('O', 0.99)]
correct = [[('I-LOC', 0.1), ('O', 0.2), ('B-LOC', 0.6), ('O', 0.7), ('I-LOC', 0.9), ('O', 0.9), ('O', 0.99), ('O', 0.99)]]
batches = [a, b]
master = map_batches(batches, blen)
print('mapped: ', master)
print('correct:', correct)

The function map_batches should do the trick. The input lists overlap only partially; for the first 2 items in list a, I have only one prediction (that of a itself). Then for the next 6 tokens, I have two predictions, corresponding to a[2:] and b[:6]. And for the last 2, I have only b again. The function is supposed to put this together, based on which has the highest likelihood (made up numbers here, in the real code these are of course confidence scores of the label).
Checking this for my toy examples a and b, it works, i.e. the output of the above is:

mapped:  [[('I-LOC', 0.1), ('O', 0.2), ('B-LOC', 0.6), ('O', 0.7), ('I-LOC', 0.9), ('O', 0.9), ('O', 0.99), ('O', 0.99)]]
correct: [[('I-LOC', 0.1), ('O', 0.2), ('B-LOC', 0.6), ('O', 0.7), ('I-LOC', 0.9), ('O', 0.9), ('O', 0.99), ('O', 0.99)]]

Sorry for the long intro.
Now here's the question(s):
First of all, this is put together rather quickly. While it works for my toy example, I'm a bit anxious that it may not be correct (hence the comments for debugging/testing) for all cases (different batch sizes, etc.), and I plan to debug it more, obviously. Still, any thought or input on this problem would be much welcomed!
Second, and I guess the real question: this is a problem that I suspect must have been solved by someone long before already (and probably in a much better and more efficient way). Would anyone know any implementations for this in keras, tensorflow, numpy, sklearn or any of the other usual suspects?


title: Python list.append(another_list) is just repeating another_list again and again

body:
I am writing a program which goes through a loop.
In the loop body it make a Python list name value and append this value to another global list values
But i am having a issue that after using

values.append(value)

but it append the value to every element of values

# values
[['closed_eye_0003.jpg_face_2.jpg', 0]]
# value
['closed_eye_0007.jpg_face_1.jpg', 0]
# after appending the value to values the output is
[['closed_eye_0007.jpg_face_1.jpg', 0], ['closed_eye_0007.jpg_face_1.jpg', 0]]

The Code is

import face_recognition
values = list()
value = list()
root_dir = '/content/dataset_facialImages_300/test/CloseFace'
isOpen = 0
for img_name in imgs_names:
  img_file = root_dir + '/' + img_name
  # Load the jpg file into a numpy array
  image = face_recognition.load_image_file(img_file)

  # Find all facial features in all the faces in the image
  face_landmarks_list = face_recognition.face_landmarks(image)
  if len(face_landmarks_list):
    # print(len(face_landmarks_list))
    # print(img_name)
    first_face = face_landmarks_list[0]

    left_eye = first_face['left_eye']
    right_eye = first_face['right_eye']

    value.clear()
    value.append(img_name)
    value.append(isOpen)
    # for i in right_eye:
    #   value.append(i[0])
    #   value.append(i[1])
    print(value)
    # values.insert( len(values), value)
    values.append(value)
    print(values)
  # print( (len(values) / len(imgs_names)) * 100 )

The Output is

['closed_eye_0003.jpg_face_2.jpg', 0]
[['closed_eye_0003.jpg_face_2.jpg', 0]]
['closed_eye_0007.jpg_face_1.jpg', 0]
[['closed_eye_0007.jpg_face_1.jpg', 0], ['closed_eye_0007.jpg_face_1.jpg', 0]]
['closed_eye_0009.jpg_face_1.jpg', 0]
[['closed_eye_0009.jpg_face_1.jpg', 0], ['closed_eye_0009.jpg_face_1.jpg', 0], ['closed_eye_0009.jpg_face_1.jpg', 0]]
['closed_eye_0012.jpg_face_1.jpg', 0]
[['closed_eye_0012.jpg_face_1.jpg', 0], ['closed_eye_0012.jpg_face_1.jpg', 0], ['closed_eye_0012.jpg_face_1.jpg', 0], ['closed_eye_0012.jpg_face_1.jpg', 0]]

If i use

values.extend(value)

it does not repeat

if i use

values.insert( len(values), value)

the output is same

The output should be

['closed_eye_0003.jpg_face_2.jpg', 0]
[['closed_eye_0003.jpg_face_2.jpg', 0]]
['closed_eye_0007.jpg_face_1.jpg', 0]
[['closed_eye_0003.jpg_face_1.jpg', 0], ['closed_eye_0007.jpg_face_1.jpg', 0]]


title: CNN with Sparse Input Does NOT Converge

body:
I am practicing supervised classify problem with CNN network

The simple NN structure is: 5(Height) * 72(Width) input --> 5 * 4 kernel --> 1 * 2 pool --> Relu ==> 1 * 3 kernel --> Relu ==> fc --> 42 output ==> softmax ==> 1 max output

Adam optimizer, learning_rate = 0.001.

The original input is rather sparse, the ratio of (pixel value == 0) > 2/3.

Following some instruction, I tried to overfit the NN with small dataset.

The NN worked well with 256 samples (accuracy = 0.95), while failed to converge with 1024 samples. In 1024-sample case, the 2nd Convolution-layer generated all-zero output.

I've also tried (3 * 3) kernels, which generated similar result.

What should I do next to detect the problem? Did the problem rooted in sparse input that CNN is bad with?

Thank you very much!


title: Sharing parameters in different nn.Moules in pytorch

body:
I've got the model that you can see below, but I need to create two instances of them that shares x2h and h2h.
Does anyone know how to do it?

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size
        self.x2h = nn.Linear(input_size, hidden_size)
        self.h2h = nn.Linear(hidden_size, hidden_size)
        self.h2o = nn.Linear(hidden_size, output_size)

        #self.softmax = nn.LogSoftmax(dim=1)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, input, hidden):

        hidden1 = self.x2h(input)
        hidden2 = self.h2h(hidden)
        hidden = hidden1 + hidden2
        output = self.h2o(hidden)
        output = self.softmax(output)

        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)


title: How to fix in-place operation in recurrent cycle Pytorch

body:
So, when I'm calling

for data, label in train_loader:
      a, cutoff = model(torch.t(data),R,1)
      loss = (cutoff-label)**2
      loss.backward()

I have this

RuntimeError: one of the variables needed for gradient computation
has been modified by an inplace operation: [torch.FloatTensor [1, 1]],
which is output 0 of SelectBackward, is at version 51;
expected version 50 instead. Hint: the backtrace further above shows
the operation that failed to compute its gradient. The variable
in question was changed in there or anywhere later. Good luck!

There is my Net forward function

 def forward(self, x,R,L):
          a = torch.zeros(R,L,1)
          for cycle in range (R):
           if cycle ==0:
               a[0]=(self.w*torch.mm(self.wih,x))

           a[cycle]= relu(torch.mm(self.whh,a[cycle-1])+ (self.w*torch.mm(self.wih,x)))
          cutoff =(sqr(torch.norm(torch.mm(torch.t(self.wih),a[R-1])))/sqr(torch.norm(x)))
          return a, cutoff

And problem is in

a[cycle] = .....a[cycle-1]....

And that cutoff is using a[R-1]
How can I fix this in-place operation or cycle?

Traceback (most recent call last):

  File "C:\Users\A877\untitled2.py", line 69, in <module>
    loss.backward()

  File "C:\Users\A877\.conda\envs\test\lib\site-packages\torch\tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)

  File "C:\Users\A877\.conda\envs\test\lib\site-packages\torch\autograd\__init__.py", line 97, in backward
    Variable._execution_engine.run_backward(


title: how to initialize Tensor for torch.cat

body:
import torch

#Y_pred = ?

for xi in X_iter:
    y_pred = net(xi).argmax(dim=1)
    Y_pred = torch.cat([Y_pred, y_pred])

How do you initialize this tensor, or is there a better way to write it？


title: Setting numpy array to slice without any in-place operations

body:
How can I do this operation efficiently without any inplace operations?

n_id = np.random.choice(np.arange(2708), size=100)
z = np.random.rand(100, 64)
z_sparse = np.zeros((2708,64))
z_sparse[n_id[:100]] = z

Essentially I want the n_id rows of z_sparse to contain z's rows, but I can't do any inplace operations because my end goal is to use this in a pytorch problem.

One though would be to create zero rows within z precisely so that the rows of z end up in the positions n_id, but not sure how this would work efficiently.

Essentially row 1 of z should be placed at row n_id[0] of z_sparse, then row 2 of z should be at row n_id[1] of z_sparse, and so on...

Here's the PyTorch error jic you are curious:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation


title: Tensor Size Miss match on loss function

body:
1:
when attempting to perfrom a pytorch training sequence using batch sizes, my loss function appears to error when the nn output and a batch are put through a MSEloss function.
2:
have tried to search about nn padding, however this is not a covnet but rather an autoencoder, similar stack over flow issues have not yielded results.
3:
the NN:
class Net(nn.Module):
    def __init__(self, input_dim=10):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, int(0.75 * input_dim))
        self.fc2 = nn.Linear(int(0.75 * input_dim), int(0.5 * input_dim))
        self.fc3 = nn.Linear(int(0.5 * input_dim), int(0.33 * input_dim))
        self.fc4 = nn.Linear(int(0.33 * input_dim), int(0.25 * input_dim))
        self.fc5 = nn.Linear(int(0.25 * input_dim), int(0.33 * input_dim))
        self.fc6 = nn.Linear(int(0.33 * input_dim), int(0.5 * input_dim))
        self.fc7 = nn.Linear(int(0.5 * input_dim), int(0.75 * input_dim))
        self.fc8 = nn.Linear(int(0.75 * input_dim), input_dim)

    def forward(self, x):
        x = torch.tanh(self.fc1(x))
        x = torch.tanh(self.fc2(x))
        x = torch.tanh(self.fc3(x))
        x = torch.tanh(self.fc4(x))
        x = torch.tanh(self.fc5(x))
        x = torch.tanh(self.fc6(x))
        x = torch.tanh(self.fc7(x))
        x = self.fc8(x)
        return torch.softmax(x, dim=1)

the train method:
def train(net, x_train, x_opt, BATCH_SIZE, EPOCHS, input_dim):
    outputs = 0
    mse = 0
    optimizer = optim.SGD(net.parameters(), lr=0.001)
    loss_function = nn.MSELoss()

    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(x_train), BATCH_SIZE)):
            batch_x = x_train[i:i + BATCH_SIZE]
            # print("bx", batch_x.size())

            batch_y = x_opt[i:i + BATCH_SIZE]
            # print("by", batch_y.size())
            net.zero_grad()
            # batch_x.view(batch_y.shape[0])
            outputs = net(batch_x)
            # print('out', outputs)

            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()  # Does the update

        print(f"Epoch: {epoch}. Loss: {loss}")

error:
 99%|█████████▉| 1452/1466 [00:02<00:00, 718.09it/s]B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\loss.py:431: UserWarning: Using a target size (torch.Size([39, 10])) that is different to the input size (torch.Size([38, 10])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
100%|█████████▉| 1465/1466 [00:02<00:00, 718.36it/s]
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "B:\tools and software\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "B:\tools and software\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/pytorch_conversion.py", line 154, in <module>
    input_dim=input_dim)
  File "B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/pytorch_conversion.py", line 64, in train
    loss = loss_function(outputs, batch_y)
  File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\loss.py", line 431, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\functional.py", line 2215, in mse_loss
    expanded_input, expanded_target = torch.broadcast_tensors(input, target)
  File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\functional.py", line 52, in broadcast_tensors
    return torch._C._VariableFunctions.broadcast_tensors(tensors)
RuntimeError: The size of tensor a (38) must match the size of tensor b (39) at non-singleton dimension 0


title: Creating a criterion that measures the F1 Loss

body:
I am currently creating criterion to measure the MSE loss function using:
loss_fcn = torch.nn.MSELoss()

loss = loss_fcn(logits[getMaskForBatch(subgraph)], labels.float())

Now I need to change it to F1 score but I cannot seem to find one library that could be used for it


title: How can I use my own data to test this Convolutional Neural Network on PyTorch?

body:
so recently I have been following this tutorial from sentdex on convolutional neural networks and I have been trying to implement his code to test the trained neural network with my own images (in this case, I just pick random pictures from the dataset used in his program). So my intention is to train the neural network, test it and finally save it so I can later load it on a separate python file to use the already trained NN on one single image.
The dataset he uses is "dogs vs cats from microsoft". This is the code where I wrote the neural network program ("main.py").
import cv2
import numpy as np
from tqdm import tqdm
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

REBUILD_DATA = False # set to true to one once, then back to false unless you want to change something in your training data.

class DogsVSCats():
    IMG_SIZE = 100
    CATS = "PetImages/Cat"
    DOGS = "PetImages/Dog"
    TESTING = "PetImages/Testing"
    LABELS = {CATS: 0, DOGS: 1}
    training_data = []

    catcount = 0
    dogcount = 0

    def make_training_data(self):
        for label in self.LABELS:
            print(label)
            for f in tqdm(os.listdir(label)):
                if "jpg" in f:
                    try:
                        path = os.path.join(label, f)
                        img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
                        img = cv2.resize(img, (self.IMG_SIZE, self.IMG_SIZE))
                        self.training_data.append([np.array(img), np.eye(2)[self.LABELS[label]]])  # do something like print(np.eye(2)[1]), just makes one_hot
                        #print(np.eye(2)[self.LABELS[label]])

                        if label == self.CATS:
                            self.catcount += 1
                        elif label == self.DOGS:
                            self.dogcount += 1

                    except Exception as e:
                        pass
                        #print(label, f, str(e))

        np.random.shuffle(self.training_data)
        np.save("training_data.npy", self.training_data)
        print('Cats:',dogsvcats.catcount)
        print('Dogs:',dogsvcats.dogcount)

class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv2d(1, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv2d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(50, 50).view(-1, 1, 50, 50)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 2) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        # max pooling over 2x2
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

net = Net()

print(net)

if REBUILD_DATA:
    dogsvcats = DogsVSCats()
    dogsvcats.make_training_data()

training_data = np.load("training_data.npy", allow_pickle=True)
print(len(training_data))

optimizer = optim.Adam(net.parameters(), lr=0.001)
loss_function = nn.MSELoss()

X = torch.Tensor([i[0] for i in training_data]).view(-1,50,50)
X = X/255.0
y = torch.Tensor([i[1] for i in training_data])

VAL_PCT = 0.1  # lets reserve 10% of our data for validation
val_size = int(len(X)*VAL_PCT)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]

BATCH_SIZE = 100
EPOCHS = 1

def train(net):
    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(train_X), BATCH_SIZE)): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
            #print(f"{i}:{i+BATCH_SIZE}")
            batch_X = train_X[i:i+BATCH_SIZE].view(-1, 1, 50, 50)
            batch_y = train_y[i:i+BATCH_SIZE]

            net.zero_grad()

            outputs = net(batch_X)
            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()    # Does the update

        print(f"Epoch: {epoch}. Loss: {loss}")

def test(net):
    correct = 0
    total = 0
    with torch.no_grad():
        for i in tqdm(range(len(test_X))):
            real_class = torch.argmax(test_y[i])
            net_out = net(test_X[i].view(-1, 1, 50, 50))[0]  # returns a list,
            predicted_class = torch.argmax(net_out)

            if predicted_class == real_class:
                correct += 1
            total += 1

    print("Accuracy: ", round(correct/total, 3))

train(net)
test(net)

PATH = './object_detection.pth'
torch.save(net.state_dict(), PATH)

After training the neural network, I want to load it in this next program and simply test the images on the NN. However, every time I run this program, the neural network is trained and tested again, which makes this process much longer and annoying. And also, I think when I run this program and then input the image into the NN, the whole "main.py" is being run.
Please, can someone help me with this? It would be amazing, as I am using this as a base to my Bachelor's thesis. Potentially I would also like to adapt this code to run my own entire dataset through it, it would be incredible if someone would help me do this, as I am a newbie on pytorch.
import cv2
from main import Net, train, test
import numpy as np

classes = ('cat', 'dog')

imsize = 50

net = Net()
net.load_state_dict(torch.load('./object_detection.pth'))

def image_loader(image_name):
    image = cv2.imread(image_name, cv2.IMREAD_GRAYSCALE)
    image = cv2.resize(image, (imsize, imsize))
    image = np.array(image)
    image = torch.Tensor(image)/255
    image = image.view(-1, 1, 50, 50)
    return image

test_image = image_loader("./PetImages/Cat/1021.jpg")

result = net(test_image)
_, predicted = torch.max(result, 1)

print(result)
print(classes[predicted[0]])


title: Image to Tensor data

body:
Library: pytorch_java_only-1.6.0
I want to convert BufferedImage/File to Tensor data, is there some method/library for this?
Python solution:
image = Image.open(image_path)
image = image.convert('RGB')
transform = Compose([ToTensor()])
image = transform(image)
image = image.view(1, 3, 64, 64).cuda()
output = my_model(image)
output = output.view(-1, self.quantity)
output = nn.functional.softmax(output, dim=1)
output = torch.argmax(output, dim=1)
output = output.view(-1, self.size)[0]

I checked out the java-demo
And that's what interests me:
Module mod = Module.load("demo-model.pt1");
Tensor data =
Tensor.fromBlob(
    new int[]  {1, 2, 3, 4, 5, 6}, // **data array**
    new long[] {2, 3} // **shape array**
    );
IValue result = mod.forward(IValue.from(data), IValue.from(3.0));
Tensor output = result.toTensor();
System.out.println("shape: " + Arrays.toString(output.shape()));
System.out.println("data: " + Arrays.toString(output.getDataAsFloatArray()));

With the array of the shape everything is clear, this is HEIGHT, WIDTH, CHANNELS of image.
But what about data array? How i can convert image to right array? Maybe i should use javacv/nd4j or something like that?
Thanks for reply!


title: Depthwise convolution to normal convolution confusion

body:
I have the task in hand to convert depthwise separable convolution block(left block in image) to full convolution block(right block in image) But I am finding it difficult in understanding what should be the output channels after first block. I understand that if we remove groups parameter, we achieve the functionality of full convolutions. Can somebody help me?
Thank you.


title: Matrix Multiplication with PyTorch

body:
I'm sorry if this is a basic question, but I am reasonably new to Pytorch and have been trying to perform linear regression. I've been stuck on the following problem for about 3 hours and have tried everything, so I hope you're able to help.
I want to predict grades using a set of four inputs (time studied, travel time, failures and absenses). For my outputs I have grades on 3 tests. There are 395 students in the dataset.
I've listed all my code at the bottom, but I'll paste the relevant part here too:
So far I've created input and output tensors and have created a model for matrix multiplication. Here's my code:
w = torch.randn(395,4, requires_grad=True)
b = torch.randn(4, requires_grad=True)
print(w)
print(b)

def model(x):
    return x @ w.t() + b

predictions = model(inputs)
print(predictions)

I know this isn't the linear regression yet, but I'm really struggling with the matrix multiplication aspect of it. Whenever I run the print(predictions) code I get the following message:
RuntimeError                              Traceback (most recent call last)
<ipython-input-277-c6db141153ac> in <module>
----> 1 preds = model(inputs)
      2 print(preds)

<ipython-input-276-3b94bfbc599e> in model(x)
      1 def model(x):
----> 2     return x @ w.t() + b

RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #3 'mat2' in call to _th_addmm_out

I have a feeling the numbers for w and b are wrong (395,4) and (4), but I have no idea why or what to change them to. Is there any chance someone could point me in the right direction please?
Thanks in advance!
Here's my whole code:
'''
import torch
import numpy as np
from numpy import genfromtxt

data = np.genfromtxt('student-mat.csv', delimiter=',',dtype=float)
data

travel_time = data[1:, 0:1]
study_time = data[1:, 1:2]
failures = data[1:, 2:3]
absenses = data[1:, 3:4]
grade_one = data[1:, 4:5]
grade_two = data[1:, 5:6]
grade_three = data[1:, 6:7]
data_input = data[1:, 0:4]
output = data[1:, 4:7]

inputs = torch.from_numpy(data_input)
outputs = torch.from_numpy(grade_one)
print(inputs)
print(grade_one)

w = torch.randn(395,4, requires_grad=True)
b = torch.randn(4, requires_grad=True)
print(w)
print(b)

def model(x):
    return x @ w.t() + b

preds = model(inputs)
print(preds)

Dataset - https://archive.ics.uci.edu/ml/datasets/Student+Performance


I have string of a sequential model saved in a file:
str(base_model)
Out[2]: 'Sequential(\n  (fc1;l1): Linear(in_features=1, out_features=15, bias=True)\n  (bn1;l1): BatchNorm1d(15, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)\n  (relu1): ReLU()\n  (fc2;l1): Linear(in_features=15, out_features=15, bias=True)\n  (bn2;l1): BatchNorm1d(15, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)\n  (relu2): ReLU()\n  (fc3;l1): Linear(in_features=15, out_features=15, bias=True)\n  (bn3;l1): BatchNorm1d(15, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)\n  (relu3): ReLU()\n  (fc4;final;l2): Linear(in_features=15, out_features=1, bias=True)\n)'

and I also have the weights somewhere else where I can easily load as a dict with the standard:
# https://pytorch.org/tutorials/beginner/saving_loading_models.html

# Saving & Loading Model for Inference
# Save/Load state_dict (Recommended)
# Save:
torch.save(model.state_dict(), PATH)

# Load:
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()

Thus, how do I create an actual python object from the sequential string?
I tried using the (probably dangerous eval function) but it didn't work since the string isn't valid python.
base_model
Python 3.8.2 (default, Mar 26 2020, 10:43:30)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.16.1
Out[1]:
Sequential(
  (fc1_l1): Linear(in_features=1, out_features=15, bias=True)
  (bn1_l1): BatchNorm1d(15, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
  (relu1): ReLU()
  (fc2_l1): Linear(in_features=15, out_features=15, bias=True)
  (bn2_l1): BatchNorm1d(15, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
  (relu2): ReLU()
  (fc3_l1): Linear(in_features=15, out_features=15, bias=True)
  (bn3_l1): BatchNorm1d(15, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
  (relu3): ReLU()
  (fc4_final_l2): Linear(in_features=15, out_features=1, bias=True)
)
eval(str(base_model))
Traceback (most recent call last):
  File "/Users/brando/anaconda3/envs/my-env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-c53f3ae3ac55>", line 1, in <module>
    eval(str(base_model))
  File "<string>", line 2
    (fc1_l1): Linear(in_features=1, out_features=15, bias=True)
            ^
SyntaxError: invalid syntax

After trying that I assumed that most likely there is a standard way to do that isn't this hacky.
Is it possible to do it?

crossposted:

https://discuss.pytorch.org/t/how-does-load-a-sequential-model-from-a-string/97648
https://www.reddit.com/r/pytorch/comments/j20d16/how_does_one_load_a_sequential_model_from_a/
How does one load a sequential model from a string in Pytorch?


title: Use CrossEntropyLoss with LogSoftmax

body:
From the Pytorch documentation, CrossEntropyLoss combines LogSoftMax and NLLLoss together in one single class
But I am curious; what happens if we use both CrossEntropyLoss for criterion and LogSoftMax in my classifier:
model_x.fc = nn.Sequential (nn.Linear(num_ftrs, 2048, bias=True), nn.ReLU(),
                               nn.Linear(2048, 1024 ), nn.ReLU(),
                               nn.Linear(1024 ,256), nn.ReLU(),
                                nn.Linear(256 ,128), nn.ReLU(),
                               nn.Linear(128, num_labels),nn.LogSoftmax(dim = 1))

criterion = nn.CrossEntropyLoss()

Then if i have saved a trained model using the code above, how can I check the criterion used by the saved model?


title: How to index intermediate dimension with an index tensor in pytorch?

body:
How can I index a tensor t with n dimensions with an index tensor of m < n dimensions, such that the last dimensions of t are preserved? The index tensor is shaped equal to tensor t for all dimensions before dimension m. Or in other terms, I want to index intermediate dimensions of a tensor, while keeping all the following dimensions of the selected indices preserved.
For example, lets say we have the two tensors:
t = torch.randn([3, 5, 2]) * 10
index = torch.tensor([[1, 3],[0,4],[3,2]]).long()

with t:
tensor([[[ 15.2165,  -7.9702],
         [  0.6646,   5.2844],
         [-22.0657,  -5.9876],
         [ -9.7319,  11.7384],
         [  4.3985,  -6.7058]],

        [[-15.6854, -11.9362],
         [ 11.3054,   3.3068],
         [ -4.7756,  -7.4524],
         [  5.0977, -17.3831],
         [  3.9152, -11.5047]],

        [[ -5.4265, -22.6456],
         [  1.6639,  10.1483],
         [ 13.2129,   3.7850],
         [  3.8543,  -4.3496],
         [ -8.7577, -12.9722]]])

Then the output I would like to have would have shape (3, 2, 2) and be:
tensor([[[  0.6646,   5.2844],
         [ -9.7319,  11.7384]],
        [[-15.6854, -11.9362],
         [  3.9152, -11.5047]],
        [[  3.8543,  -4.3496],
         [ 13.2129,   3.7850]]])

Another example would be that I have a tensor t of shape (40, 10, 6, 2) and an index tensor of shape (40, 10, 3). This should query dimension 3 of tensor t and the expected output shape would be (40, 10, 3, 2).
How can I achieve this in a generic way, without using loops?


Here's my evaluation cell:
start_time = time.time()

with torch.no_grad():

    best_network = Network()
    best_network.cuda()
    best_network.load_state_dict(torch.load('../moth_landmarks.pth'))
    best_network.eval()

    batch = next(iter(train_loader))
    images, landmarks = batch['image'], batch['landmarks']
    #images = images.unsqueeze_(1)

    images = torch.cat((images,images,images),1)
    images = images.cuda()

    norm_image = transforms.Normalize(0.3812, 0.1123)
    for image in images:
        image = image.float()
        ##image = to_tensor(image) #TypeError: pic should be PIL Image or ndarray. Got <class 'torch.Tensor'>
        image = norm_image(image)
    landmarks = (landmarks + 0.5) * 224

    ##[8, 600, 800] --> [8,3,600,800]
    images = images.unsqueeze(1)
    images = torch.cat((images, images, images), 1)

    predictions = (best_network(images).cpu() + 0.5) * 224
    predictions = predictions.view(-1,4,2)

    plt.figure(figsize=(10,40))

    for img_num in range(8):
        plt.subplot(8,1,img_num+1)
        plt.imshow(images[img_num].cpu().numpy().transpose(1,2,0).squeeze(), cmap='gray')
        plt.scatter(predictions[img_num,:,0], predictions[img_num,:,1], c = 'r')
        plt.scatter(landmarks[img_num,:,0], landmarks[img_num,:,1], c = 'g')

print('Total number of test images: {}'.format(len(test_dataset)))

end_time = time.time()
print("Elapsed Time : {}".format(end_time - start_time))

How should I fix the following error?
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-59-e4aa0ace8c75> in <module>
     19         image = image.float()
     20         ##image = to_tensor(image) #TypeError: pic should be PIL Image or ndarray. Got <class 'torch.Tensor'>
---> 21         image = norm_image(image)
     22     landmarks = (landmarks + 0.5) * 224
     23

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, tensor)
    210             Tensor: Normalized Tensor image.
    211         """
--> 212         return F.normalize(tensor, self.mean, self.std, self.inplace)
    213
    214     def __repr__(self):

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
    282     if tensor.ndimension() != 3:
    283         raise ValueError('Expected tensor to be a tensor image of size (C, H, W). Got tensor.size() = '
--> 284                          '{}.'.format(tensor.size()))
    285
    286     if not inplace:

ValueError: Expected tensor to be a tensor image of size (C, H, W). Got tensor.size() = torch.Size([1800, 800]).

If I remove the following line:
images = torch.cat((images,images,images),1)

I will get this new error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-61-3e892b69015c> in <module>
     19         image = image.float()
     20         ##image = to_tensor(image) #TypeError: pic should be PIL Image or ndarray. Got <class 'torch.Tensor'>
---> 21         image = norm_image(image)
     22     landmarks = (landmarks + 0.5) * 224
     23

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, tensor)
    210             Tensor: Normalized Tensor image.
    211         """
--> 212         return F.normalize(tensor, self.mean, self.std, self.inplace)
    213
    214     def __repr__(self):

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
    282     if tensor.ndimension() != 3:
    283         raise ValueError('Expected tensor to be a tensor image of size (C, H, W). Got tensor.size() = '
--> 284                          '{}.'.format(tensor.size()))
    285
    286     if not inplace:

ValueError: Expected tensor to be a tensor image of size (C, H, W). Got tensor.size() = torch.Size([600, 800]).


title: How to generate an onnx file with linear layers using Pytorch

body:
I want to create a network on the basis of the vgg16 network, but adding linear layers (Gemm) just after the conv2d layers, for normalization purpose.
After that, I want to export the network in an ONNX file.
The first part seems to work: I took the Pytorch code for generating the vgg16 and modified it as follows
import torch.nn as nn

class VGG(nn.Module):

    def __init__(self, features, num_classes=8, init_weights=True):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.Linear(4096, 4096),  # New shift layer
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.Linear(4096, 4096),  # New shift layer
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 8),
            nn.Linear(8, 8),        # New shift layer
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

def make_layers(cfg, batch_norm=False):
    layers = []
    in_channels = 3
    n = 224
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            n = int(n / 2)
        elif v == 'B':
            layers += [nn.AdaptiveAvgPool2d(n)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            linear = nn.Linear(n,n,True)
            if batch_norm:
                layers += [conv2d, linear, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, linear, nn.ReLU(inplace=True)]
            in_channels = v
    return nn.Sequential(*layers)

cfg = {'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M','B'],
}

def vgg16(**kwargs):
    """VGG 16-layer model (configuration "D")

    """
    model = VGG(make_layers(cfg['D']), **kwargs)
    return model

But when I insert the weights and export to onnx, I see that my linear layers are not referred to as Gemm but as {Transpose + Matmult + Add}

The Transpose part is the weights matrix and the Add part is for the biases (which are all 0).
Am I wrong to think that it's possible to do this, or is there a way to get a real Gemm layer here or another way to do this normalization (which is simply multiply all outputs by a single value)?


title: BERT DataLoader: Difference between shuffle=True vs Sampler?

body:
I trained a DistilBERT model with DistilBertForTokenClassification on ConLL data fro predicting NER. Training seem to have completed with no problems but I have 2 problems during evaluation phase.

I'm getting negative loss value

During training, I used shuffle=True for DataLoader. But during evaluation, when I do shuffle=True for DataLoader, I get very poor metric results(f_1, accuracy, recall etc). But if I do shuffle = False or use a Sampler instead of shuffling I get pretty good metric results. I'm wondering if there is anything wrong with my code.

Here is the evaluation code:

print('Prediction started on test data')
model.eval()

eval_loss = 0
predictions , true_labels = [], []

for batch in val_loader:
  b_input_ids = batch['input_ids'].to(device)
  b_input_mask = batch['attention_mask'].to(device)
  b_labels = batch['labels'].to(device)

  with torch.no_grad():
      outputs = model(b_input_ids,
                      attention_mask=b_input_mask)

  logits = outputs[0]
  logits = logits.detach().cpu().numpy()
  label_ids = b_labels.detach().cpu().numpy()

  predictions.append(logits)
  true_labels.append(label_ids)

  eval_loss += outputs[0].mean().item()

print('Prediction completed')
eval_loss = eval_loss / len(val_loader)
print("Validation loss: {}".format(eval_loss))

out:
Prediction started on test data
Prediction completed
Validation loss: -0.2584906197858579

I believe I'm calculating the loss wrong here. Is it possible to get negative loss values with BERT?
For DataLoader, if I use the code snippet below, I have no problems with the metric results.
val_sampler = SequentialSampler(val_dataset)
val_loader = DataLoader(val_dataset, sampler=val_sampler, batch_size=128)

Bu if I do this one I get very poor metric results
val_loader = DataLoader(val_dataset, batch_size=128, shuffle=True)

Is it normal that I'm getting vastly different results with shuffle=True vs shuffle=False ?
code for the metric calculation:
metric = load_metric("seqeval")
results = metric.compute(predictions=true_predictions, references=true_labels)
results

out:
{'LOCATION': {'f1': 0.9588207767898924,
  'number': 2134,
  'precision': 0.9574766355140187,
  'recall': 0.9601686972820993},
 'MISC': {'f1': 0.8658965344048217,
  'number': 995,
  'precision': 0.8654618473895582,
  'recall': 0.8663316582914573},
 'ORGANIZATION': {'f1': 0.9066332916145182,
  'number': 1971,
  'precision': 0.8947628458498024,
  'recall': 0.9188229325215627},
 'PERSON': {'f1': 0.9632426988922457,
  'number': 2015,
  'precision': 0.9775166070516096,
  'recall': 0.9493796526054591},
 'overall_accuracy': 0.988255561629313,
 'overall_f1': 0.9324058459808882,
 'overall_precision': 0.9322748349023465,
 'overall_recall': 0.932536893886156}

The above metrics are printed when I use Sampler or shuffle=False. If I use shuffle=True, I get:
{'LOCATION': {'f1': 0.03902284263959391,
  'number': 2134,
  'precision': 0.029496402877697843,
  'recall': 0.057638238050609185},
 'MISC': {'f1': 0.010318142734307824,
  'number': 995,
  'precision': 0.009015777610818933,
  'recall': 0.012060301507537688},
 'ORGANIZATION': {'f1': 0.027420984269014285,
  'number': 1971,
  'precision': 0.019160951996772892,
  'recall': 0.04819888381532217},
 'PERSON': {'f1': 0.02119907254057635,
  'number': 2015,
  'precision': 0.01590852597564007,
  'recall': 0.03176178660049628},
 'overall_accuracy': 0.5651741788003777,
 'overall_f1': 0.02722600361161272,
 'overall_precision': 0.020301063389034663,
 'overall_recall': 0.041321152494729445}

UPDATE: I modified loss code for evaluation. There seems to be no problem with this code. You can see the new code below:
print('Prediction started on test data')
model.eval()

eval_loss = 0
predictions , true_labels = [], []

for batch in val_loader:

  b_labels = batch['labels'].to(device)

  batch = {k:v.type(torch.long).to(device) for k,v in batch.items()}

  with torch.no_grad():
      outputs = model(**batch)

      loss, logits = outputs[0:2]
      logits = logits.detach().cpu().numpy()
      label_ids = b_labels.detach().cpu().numpy()

      predictions.append(logits)
      true_labels.append(label_ids)

      eval_loss += loss

print('Prediction completed')
eval_loss = eval_loss / len(val_loader)
print("Validation loss: {}".format(eval_loss))

Though I still haven't got an asnwer to the DataLoader question.
Also I jsut realised when I do print(model.eval()) I still get dropouts from the model in evaluation mode.


title: How to transfer a PyTorch graph from AWS SageMaker to S3?

body:
I just successfully completed PyTorch training on SageMaker for the 1st time and now I'm attempting to work out how to upload the resulting graph to an S3 bucket.
For local PyTorch training I would do:
GRAPH_NAME = 'MNIST.pt'

# training and other content here

torch.save(mnistNet.state_dict(), GRAPH_NAME)

The S3 bucket I'm writing to has at it's root a directory named mnist_images and I'd like to save the graph there, so I attempted the equivalent SageMaker -> S3 upload like this:
s3 = boto3.client('s3')
s3.put_object(Body=mnistNet.state_dict(), Bucket=S3_BUCKET_NAME, Key='mnist_images/' + GRAPH_NAME)

I'm getting the error:
ParamValidationError: Parameter validation failed:
Invalid type for parameter Body, value: OrderedDict([('conv1.weight', tensor([[[[ 0.0518,  0.2257,  0.0447,  0.1843,  0.0698],
          [ 0.2228,  0.0152,  0.1453,  0.2463,  0.2031],
          [ 0.0441,  0.0754, -0.0683,  0.1884,  0.2654],
          [ 0.2718,  0.1279,  0.1234,  0.0440,  0.0778],
          [ 0.2234,  0.2384,  0.0941,  0.0526, -0.0817]]],
          .
          .
          .
          (many similar lines omitted)
          .
          .
          .

I gather before passing mnistNet.state_dict() to the Body= parameter I need to do some sort of conversion, can somebody who has done this before advise on how do to this please?
Something else I can mention is before the training process I was able to save my labels list to the same S3 location as a json file like so:
s3 = boto3.client('s3')
s3.put_object(Body=str(json.dumps(trainLabels)), Bucket=S3_BUCKET_NAME, Key='mnist_images/labels.json')

So I'm certain I have the location correct and my permissions are sufficient.
--- Edit ---
I worked out that I had to add pickling to save the graph to S3:
s3 = boto3.client('s3')
s3.put_object(Body=pickle.dumps(mnistNet.state_dict()), Bucket=S3_BUCKET_NAME, Key='mnist_images/' + GRAPH_NAME)

This successfully saves the graph from SageMaker to S3.  However, when I write an inference script in SageMaker and attempt to load:
graphObj = botoClient.get_object(Bucket=S3_BUCKET_NAME, Key='mnist_images/MNIST.pt')
graph = pickle.loads(graphObj['Body'].read())
net.load_state_dict(torch.load(graph))

I get this error which I've stuck on for quite some time now:
'collections.OrderedDict' object has no attribute 'seek'

Also I should mention this page:
https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html
Makes mention of a magic environment variable os.environ['SM_MODEL_DIR'] but this does not exist in SageMaker and I get a key error if I attempt to reference it.
Help!  There has to be a way to save (preferably to an S3 bucket) and at a later time load a PyTorch model in SageMaker.


title: Manual mini-batch generation for PyTorch Geometric

body:
Currently I have pytorch tensors with shape (batch_size, height, width, channel_size) and I want to convert it to a mini-batch described here. My current idea is to convert each example from tensor representation to graph representation separately and group them together. I want to do all these without involving file save/load as it will surely hinder the speed (I notice that Creating “In Memory Datasets” does it this way).
Yet I didn't find any function for the example grouping part. Could anyone please help give a plausible workflow for it, and is there any smarter way for this convertion, from tensor to mini-batch for pytorch-geometric?


title: The loss will be Nan when I use loss function defined by torch.nn.function.mse_loss

body:
The loss is always Nan when I use the loss function as follow:
def Myloss1(source, target):
    loss = torch.nn.functional.mse_loss(source, target, reduction="none")
    return torch.sum(loss).sqrt()

...

loss = Myloss1(s, t)
loss.backward()

But when I use the following loss function, the training becomes normal:
def Myloss2(source, target):
    diff = target - source
    loss = torch.norm(diff)
    return loss
...

loss = Myloss2(s, t)
loss.backward()

Why can't use the ‘Myloss1’ to train？ Aren't Myloss1 and Myloss2 equivalent?
Please help me，thank you very much！


title: Pretrained model or training from scratch for object detection?

body:
I have a dataset composed of 10k-15k pictures for supervised object detection which is very different from Imagenet or Coco (pictures are much darker and represent completely different things, industrial related).
The model currently used is a FasterRCNN which extracts features with a Resnet used as a backbone.
Could train the backbone of the model from scratch in one stage and then train the whole network in another stage be beneficial for the task, instead of loading the network pretrained on Coco and then retraining all the layers of the whole network in a single stage?


title: take the output from a specific layer in pytorch

body:
I have implemented an autoencoder in Pytorch and wish to extract the representations (output) from a specified encoding layer. This setup is similar to making predictions using sub-models that we used to have in Keras.
However, implementing something similar in Pytorch looks a bit challenging. I tried forward hooks as explained in How to get the output from a specific layer from a PyTorch model? and https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html but to no avail.
Could you help me getting outputs from a specific layer?
I have attached my code below:
class Autoencoder(torch.nn.Module):

    # Now defining the encoding and decoding layers.

    def __init__(self):
        super().__init__()
        self.enc1 = torch.nn.Linear(in_features = 784, out_features = 256)
        self.enc2 = torch.nn.Linear(in_features = 256, out_features = 128)
        self.enc3 = torch.nn.Linear(in_features = 128, out_features = 64)
        self.enc4 = torch.nn.Linear(in_features = 64, out_features = 32)
        self.enc5 = torch.nn.Linear(in_features = 32, out_features = 16)
        self.dec1 = torch.nn.Linear(in_features = 16, out_features = 32)
        self.dec2 = torch.nn.Linear(in_features = 32, out_features = 64)
        self.dec3 = torch.nn.Linear(in_features = 64, out_features = 128)
        self.dec4 = torch.nn.Linear(in_features = 128, out_features = 256)
        self.dec5 = torch.nn.Linear(in_features = 256, out_features = 784)

    # Now defining the forward propagation step

    def forward(self,x):
        x = F.relu(self.enc1(x))
        x = F.relu(self.enc2(x))
        x = F.relu(self.enc3(x))
        x = F.relu(self.enc4(x))
        x = F.relu(self.enc5(x))
        x = F.relu(self.dec1(x))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        x = F.relu(self.dec5(x))

        return x

autoencoder_network = Autoencoder()

I have to take the output from encoder layers marked enc1, enc2 .., enc5.


title: Shape of ground truth in multiclass image segmentation with pytorch

body:
I'm working on 128 x 128 x 3 cell images and want to segment them into 5 classes including backgrounds. I first made target images to be 128 x 128 and values are in {0,1,2,3,4}. But I found I have to make my target ground truth as 5-channel image, and all the values are 0 or 1: if a pixel has 1 in the nth channel, then it should be classified to nth class.
But when I run my model into a Unet model which I forked from GitHub, I found there's an error while calculating cross-entropy loss.
I initially set up the number of channels in the input to be 3 and the number of classes in the output to be 5. And batch size = 2
Here is my codes:
for i, (x, y) in batch_iter:
    input, target = x.to(self.device), y.to(self.device)  # send to device (GPU or CPU)
    self.optimizer.zero_grad()  # zerograd the parameters
    out = self.model(input)  # one forward pass
    loss = self.criterion(out, target)  # calculate loss
    loss_value = loss.item()
    train_losses.append(loss_value)
    loss.backward()  # one backward pass
    self.optimizer.step()  # update the parameters

    batch_iter.set_description(f'Training: (loss {loss_value:.4f})')  # update progressbar

self.training_loss.append(np.mean(train_losses))
self.learning_rate.append(self.optimizer.param_groups[0]['lr'])

batch_iter.close()

And error message
RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [2, 5, 128, 128]

How can I solve this?


title: PyTorch dataloading “freezes” when running several jobs on a GCP instance

body:
I have a problem when running several python3.7 jobs on one machine (one job per GPU).
Those are machine-learning jobs involving PyTorch dataloading for a large dataset (>1000GB).
With only one job, the training goes well if I use the pin_memory=True option of the Dataloader, the batches are provided on time, the gpu utilisation is high. I use 8 workers.
If pin_memory=False, the main process waits a bit for the dataloading on some batches, but it is not too bad.
However, if I start a second job, the dataloading “freezes” for every 8th batch (I have 8 workers) for as long as one minute, slowing down the training dramatically. I timed the __getitem__ and collate functions and timing stays short and constant for those functions. It is therefore not a problem of disk throughput or i/o, nor processing.
I found during the “freezes” that all the dataloading workers processes are dominated by Kernel Threads (all cpus are displayed in red color on htop).
Turning pin_memory=False does not solve the issue.
So if it does not come from the __getitem__ nor collate functions, where is all that time spent?
What is going on after one batch has been processed by a worker and before it can be consumed in the main process?
Could you recommend some monitoring tools to find out where the problem comes from?
I am working on a GCP linux virtual machine and my data is stored on a SSD drive.
Thanks for your help.


title: Optimize input instead of network in pytorch

body:
I am trying to optimize the inputs in some required task and I don't want to update my network as its freezed. I have written a minimal example but its not working as z has the same value in all iterations.  I am sure that I am doing some silly mistake in this process. Any guidance is highly appreciated.
thanks.
import torch

z = torch.rand((1,6))
z.requires_grad_(True)
optimizer = torch.optim.SGD([z], lr= 0.1)

criteria = torch.nn.MSELoss()

for i in range(5):
    optimizer.zero_grad()
    print(z)
    loss = criteria(z, z+torch.rand(1))
    #print(loss)
    loss.backward()
    optimizer.step()

##output
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)


title: Sequence to Sequence Loss

body:
I'm trying to figure out how sequence to sequence loss is calculated. I am using the huggingface transformers library in this case, but this might actually be relevant to other DL libraries.
So to get the required data we can do:
from transformers import EncoderDecoderModel, BertTokenizer
import torch
import torch.nn.functional as F
torch.manual_seed(42)

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
MAX_LEN = 128
tokenize = lambda x: tokenizer(x, max_length=MAX_LEN, truncation=True, padding=True, return_tensors="pt")

model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints
input_seq = ["Hello, my dog is cute", "my cat cute"]
output_seq = ["Yes it is", "ok"]
input_tokens = tokenize(input_seq)
output_tokens = tokenize(output_seq)

outputs = model(
    input_ids=input_tokens["input_ids"],
    attention_mask=input_tokens["attention_mask"],
    decoder_input_ids=output_tokens["input_ids"],
    decoder_attention_mask=output_tokens["attention_mask"],
    labels=output_tokens["input_ids"],
    return_dict=True)

idx = output_tokens["input_ids"]
logits = F.log_softmax(outputs["logits"], dim=-1)
mask = output_tokens["attention_mask"]

Edit 1
Thanks to @cronoik I was able to replicate the loss calculated by huggingface as being:
output_logits = logits[:,:-1,:]
output_mask = mask[:,:-1]
label_tokens = output_tokens["input_ids"][:, 1:].unsqueeze(-1)
select_logits = torch.gather(output_logits, -1, label_tokens).squeeze()
huggingface_loss = -select_logits.mean()

However, since the last two tokens of the second input is just padding, shouldn't we calculate the loss to be:
seq_loss = (select_logits * output_mask).sum(dim=-1, keepdims=True) / output_mask.sum(dim=-1, keepdims=True)
seq_loss = -seq_loss.mean()

^This takes into account the length of the sequence of each row of outputs, and the padding by masking it out. Think this is especially useful when we have batches of varying length outputs.


title: Why a.storage() is b.storage() returns false when a and b reference the same data?

body:
>>> a = torch.arange(12).reshape(2, 6)
>>> a
tensor([[ 0,  1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10, 11]])
>>> b = a[1:, :]
>>> b.storage() is a.storage()
False

But
>>> b[0, 0] = 999
>>> b, a # both tensors are changed
(tensor([[999,   7,   8,   9,  10,  11]]),
 tensor([[  0,   1,   2,   3,   4,   5],
         [999,   7,   8,   9,  10,  11]]))

What is exactly the objects that stores tensor data? How can I make check if 2 tensors share memory?


title: Interpreting the output tokenization of BERT for a given word

body:
from bert_embedding import BertEmbedding
bert_embedding = BertEmbedding(model='bert_12_768_12', dataset_name='wiki_multilingual_cased')
output = bert_embedding("any")

I need clarification on the output of mBERT embeddings. I'm aware that WordPiece tokenization is used to break up the input text. Also I observed that on providing a single word (say "any") as input, the output has length equal to the number of characters in the input (in our case, 3). output[i] is a tuple of lists where the first list contains the character at ith  position with the 'unknown' token preceding and following it as different elements in the array. Following this are three (= length of the input word) arrays (embeddings) of size 768 each. Why does the output seem to be tokenized character-wise (rather than wordpiece tokenized)?
Also found out the output form changes when the input is given in a list as:bert_embedding(["any"]). The output now is a single tuple with ['[UNK]', 'state', '[UNK]'] as the first element followed by three different embeddings conceivably corresponding to the three tokens listed above.
If I need the embedding of the last subword (not simply of the last character or the whole word) for a given input word, how do I access it?


title: Anaconda installed PyTorch is throwing runtime error while trying to access GPU

body:
**~\anaconda3\lib\site-packages\torch\cuda\__init__.py in _lazy_init()

168         # This function throws if there's a driver initialization error, no GPUs

169         # are found or any other error occurs

170         torch._C._cuda_init()

171         # Some of the queued calls may reentrantly call _lazy_init();

172         # we need to just return without initializing in that c**ase.

      RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu.

As seen above, I have already cheacked my gpu drivers.
My cuda version and gpu drivers:

PyTorch 1.8.1
CUDA 11.1.1
cudnn 8.0


title: Why can't I use my graphic card with PyTorch?

body:
I am trying to use PyTorch with GPU in my Ubuntu 18.04. The GPU is a GeForce GTX 1070.
nvidia-smi:
| NVIDIA-SMI 460.67       Driver Version: 460.67       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:0B:00.0 Off |                  N/A |
| 21%   49C    P2    60W / 180W |   4598MiB /  8119MiB |     17%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1070    Off  | 00000000:42:00.0 Off |                  N/A |
|  0%   48C    P8     7W / 180W |     20MiB /  8117MiB |      0%      Default |
|                               |                      |                  N/A |

nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

.bashrc file:
export PATH="/usr/local/cuda-10.2/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-10.2/lib64:$LD_LIBRARY_PATH"

I installed pytorch using the command below (from here):
pip install torch torchvision torchaudio

Torch version:
PyTorch Version: 1.8.0

Python version:
Python 3.8.7

gcc/g++ versions:
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

If I try to get the GPU, I get the following:
>>> import torch
>>> print(torch.cuda.is_available())
False

Can anyone advice me please


title: Different test results with pytorch lightning

body:
I use Pytorch Lightning to train a small NN transfert learning) with the hymenoptera photos (inspired from here).
In the test_step method, it prints the real classes (classes) and the predictions (preds).
After the training, I do the same (verification step) but I get different results.
import torch
from torch import nn
from torch.optim import Adam,SGD
import pytorch_lightning as pl
from torchvision import  models
from torch.optim import lr_scheduler
from pytorch_lightning.metrics.functional import accuracy
from pytorch_lightning.loggers import TensorBoardLogger
from hymenoptereDataModule import HymenopteraDataModule

class LitHymenoptera(pl.LightningModule):

    def __init__(self, batch_size=4):
        super().__init__()
        torch.manual_seed(42)
        self.batch_size = batch_size
        self.dataModule = HymenopteraDataModule()
        self.dataModule.setup()
        self.criterion = nn.CrossEntropyLoss()
        self.logger = TensorBoardLogger('tb_logs', name=f'Model')
        # Define the model
        self.model = models.resnet18(pretrained=True)
        num_ftrs = self.model.fc.in_features
        self.model.fc = nn.Linear(num_ftrs, 2)

    def forward(self, x):
        return self.model(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        logits = self.model(x)
        # Compute loss
        loss = self.criterion(logits, y)
        # training metrics
        preds = torch.argmax(logits, dim=1)
        acc = accuracy(preds, y)
        num_correct = torch.eq(preds.view(-1), y.view(-1)).sum()
        return {'loss': loss,
                'acc': acc,
                'num_correct': num_correct}

    def training_epoch_end(self, outputs):
        self.exp_lr_scheduler.step()

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self.model(x)
        loss = self.criterion(logits, y)
        # validation metrics
        preds = torch.argmax(logits, dim=1)
        acc = accuracy(preds, y)
        num_correct = torch.eq(preds.view(-1), y.view(-1)).sum()
        return {'loss': loss,
                'acc': acc,
                'num_correct': num_correct}

    def test_step(self, batch, batch_idx):
        inputs, classes = batch
        logits = self(inputs)
        preds = torch.argmax(logits, dim=1)
        print('###############################')
        print('classes1 = ',classes)
        print('preds1 = ',preds)
        print(logits)

    def configure_optimizers(self):
        optimizer = SGD(self.parameters(), lr=0.001, momentum=0.9)
        # Decay LR by a factor of 0.1 every 7 epochs
        self.exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
        return optimizer

model = LitHymenoptera()
trainer = pl.Trainer(gpus=1, max_epochs=5, progress_bar_refresh_rate=100)
trainer.fit(model,model.dataModule)
trainer.test(model)

# Now, another test
for inputs, classes in model.dataModule.val_dataloader():
    print('###############################')
    logits = model(inputs.cuda())
    preds = torch.argmax(logits, dim=1)
    print('classes2 = ', classes)
    print('preds2 = ', preds)
    print(logits)

Here's the first output from test_step method :
classes1 =  tensor([0, 0, 0, 0], device='cuda:0')
preds1 =  tensor([1, 0, 0, 0], device='cuda:0') tensor([[0.1626, 0.2195],
            [1.1437, 0.5745],
            [0.9351, 0.4271],
            [0.7365, 0.5342]], device='cuda:0')

and now the first output from the verification step :
classes2 =  tensor([0, 0, 0, 0])
preds2 =  tensor([1, 0, 1, 0], device='cuda:0')
tensor([[-0.0168,  0.0800],
        [ 0.6817,  0.2949],
        [-0.2205,  0.1009],
        [ 0.6126,  0.4924]], device='cuda:0', grad_fn=<AddmmBackward>)

Both classes are identical (and I check the images, they are the same) but the preds are different.
Where does it come from?


I am trying to use the roberta transformer and a pre-trained model but I keep getting this error:
    ImportError:
AutoModelForSequenceClassification requires the PyTorch library but it was not found in your environment. Checkout the instructions on the
installation page: https://pytorch.org/get-started/locally/ and follow the ones that match your environment.

Here's my code:
# Tasks:
# emoji, emotion, hate, irony, offensive, sentiment
# stance/abortion, stance/atheism, stance/climate, stance/feminist, stance/hillary

task='sentiment'
MODEL = f"cardiffnlp/twitter-roberta-base-{task}"

tokenizer = AutoTokenizer.from_pretrained(MODEL)
# download label mapping
labels=[]
mapping_link = f"https://raw.githubusercontent.com/cardiffnlp/tweeteval/main/datasets/{task}/mapping.txt"
model = AutoModelForSequenceClassification.from_pretrained(MODEL)
model.save_pretrained(MODEL)
labels=[]
mapping_link = f"https://raw.githubusercontent.com/cardiffnlp/tweeteval/main/datasets/{task}/mapping.txt"
with urllib.request.urlopen(mapping_link) as f:
    html = f.read().decode('utf-8').split("\n")
    csvreader = csv.reader(html, delimiter='\t')
labels = [row[1] for row in csvreader if len(row) > 1]

I made sure that PyTorch is installed and working:


I want to use pytorch DistributedDataParallel for adversarial training. The loss function is trades.The code can run in DataParallel mode. But in DistributedDataParallel mode, I got this error.
When I change the loss to AT, it can run successfully. Why can't run with trades loss? The two loss functions are as follows：
-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/lthpc/.conda/envs/bba/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/data/zsd/defense/my_adv_training/muti_gpu_2.py", line 170, in main_worker
    train_loss = train(train_loader, model, optimizer, epoch, local_rank, args)
  File "/data/zsd/defense/my_adv_training/muti_gpu_2.py", line 208, in train
    loss = trades(model,x, y,optimizer, args.epsilon,args.step_size,args.num_steps,beta=6.0)
  File "/data/zsd/defense/my_adv_training/loss_functions.py", line 137, in trades
    loss_kl.backward()
  File "/home/lthpc/.conda/envs/bba/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/lthpc/.conda/envs/bba/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [640]] is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

.
for i, (x, y) in enumerate(train_loader):
    # measure data loading time
    x,y = x.cuda(local_rank, non_blocking=True),  y.cuda(local_rank, non_blocking=True)

    loss = trades(model,x, y,optimizer, args.epsilon,args.step_size,args.num_steps,beta=6.0)
    torch.distributed.barrier()

    optimizer.zero_grad()
    loss.backward(retain_graph=True)
    optimizer.step()

def trades():
    model.eval()
    criterion_kl = nn.KLDivLoss(reduction='sum')
    x_adv = x.detach() + 0.001 * torch.randn_like(x).detach()
    nat_output = model(x)
    for _ in range(num_steps):
        x_adv.requires_grad_()
        with torch.enable_grad():
            loss_kl = criterion_kl(F.log_softmax(model(x_adv), dim=1),
                               F.softmax(nat_output, dim=1))
        loss_kl.backward()
        eta = step_size * x_adv.grad.sign()
        x_adv = x_adv.detach() + eta
        x_adv = torch.min(torch.max(x_adv, x - epsilon), x + epsilon)
        x_adv = torch.clamp(x_adv, 0.0, 1.0)
    model.train()
    x_adv = Variable(x_adv, requires_grad=False)
    optimizer.zero_grad()
    # calculate robust loss
    logits = model(x)
    loss_natural = nn.CrossEntropyLoss()(logits, y)
    loss_robust = (1.0 / x.size(0)) * criterion_kl(F.log_softmax(model(x_adv), dim=1),
                           F.softmax(logits, dim=1))
    loss = loss_natural + beta * loss_robust
    return loss

def AT():
    model.eval()
    x_adv = x.detach() + torch.from_numpy(np.random.uniform(-epsilon,
epsilon, x.shape)).float().cuda()
    x_adv = torch.clamp(x_adv, 0.0, 1.0)
    for k in range(num_steps):
        x_adv.requires_grad_()
        output = model(x_adv)
        model.zero_grad()
        with torch.enable_grad():
            loss = nn.CrossEntropyLoss()(output, y)
        loss.backward()
        eta = step_size * x_adv.grad.sign()
        x_adv = x_adv.detach() + eta
        x_adv = torch.min(torch.max(x_adv, x - epsilon), x + epsilon)
        x_adv = torch.clamp(x_adv, 0.0, 1.0)
    x_adv = Variable(x_adv, requires_grad=False)

    model.train()
    logits_adv = model(x_adv)
    loss = nn.CrossEntropyLoss()(logits_adv, y)
    return loss


I'm using python 3.7 in google colab. I install allennlp=2.4.0 and allennlp-models.
When I run my code, I get this error:
from allennlp.data.tokenizers.word_splitter import SpacyWordSplitter

ModuleNotFoundError: No module named
'allennlp.data.tokenizers.word_splitter'


title: Train a neural network when the training has only the derivative of output wrt all inputs

body: There is a scalar function F with 1000 inputs. I want to train a model to predict F given the inputs. However, in the training dataset, we only know the derivative of F with respect to each input, not the value of F itself. How I can construct a neural network with this limitation in tensorflow or pytorch?


title: Is Torch7 defined-by-run like Pytorch?

body:
Pytorch have Dynamic Neural Networks (defined-by-run) as opposed to Tensorflow which have to compile the computation graph before run.

I see that both Torch7 and PyTorch depend on TH, THC, THNN, THCUNN (C library). Does Torch7 have Dynamic Neural Networks (defined-by-run) feature ?


title: Slicing a numpy 3-d matrix into 2-d matrix

body:
I have a 3d numpy matrix t as follows, generated randomly:

t = np.random.rand(2,2,2)
array([[[ 0.80351862,  0.25631294],
        [ 0.7971346 ,  0.29468456]],

       [[ 0.33771957,  0.91776256],
        [ 0.6018604 ,  0.55290615]]])

I want to extract a 2-d matrix such that the result is sliced along the columns of the 3-d matrix. Something like:

array([[ 0.25631294  ,  0.91776256],
       [ 0.29468456,  0.55290615]])

How can I slice in such a way?

Thanks for the help.


title: how to select values from tensor contained in np.array

body:
I have this huge tensor from which I just want to keep selected tensors.

Background - first contains coordinates of quadrilaterals being predicted.

np.shape(coords_detached) =  (15969, 8)
coords.shape() =   torch.Size([15969, 8])

The second, contains same coordinates but filtered after selection using NMS, for this discussion just say I select 9 rows from above tensor. 8 coordinates + 1 confidence score
But NMS is being done in numpy so I detach the tensors.

coords_nms = torch.tensor(nms_coords_, dtype=torch.float32)

coords_nms.shape() =  torch.Size([9, 9])

So now I want to select just these 9 rows from the original tensor, coz it had the gradient information that gets lost during detach() and numpy nms.

I tried this :

s = torch.ones_like(nms_coords_)
s *=-1

nms_coords = torch.where(coords == coords_nms[:,:-1], coords, s)
nms_coords = nms_coords[nms_coords>=0]
nms_coords.reshape(-1, 8)

to iterate through coords and match value coords_nms and just store those. but it needs same dimension at axis=0

The iterative loop would be the following but how to do it using tensor notation :

poo = []
for x in coords:
    for z in nms_coords_:
       if sum(x[:] == z[:-1]) == 8 :
            poo.append(z[:-1])


title: Can I implement EdgespeechNets (google) using pytorch-kaldi?

body: I would like to implement EdgespeechNets , Im wondering if kaldi-pytorch is a good toolkit for that  ?


title: Loss and accuracy not changing of LSTM model

body:
Picture of the train and test lossHere’s a look at the LSTM model
`

# Create LSTM Model
class LSTMModel(nn.Module):
  def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
    super(LSTMModel, self).__init__()
    # Number of hidden dimensions
    self.hidden_dim = hidden_dim

    # Number of hidden layers
    self.layer_dim = layer_dim

    # LSTM
    self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True, dropout=0.1)

    # Readout layer
    self.f1 = nn.Linear(hidden_dim, output_dim)

    self.dropout_layer = nn.Dropout(p=0.2)
    self.softmax = nn.Softmax()
    def forward(self, x):
    # Initialize hidden state with zeros

       h0 = Variable(torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).type(torch.FloatTensor))

    # Initialize cell state

       c0 = Variable(torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).type(torch.FloatTensor))

    # One time step
       out, (hn, cn) = self.lstm(x, (h0,c0))
       out = self.dropout_layer(hn[-1])
       out = self.f1(out)
       out = self.softmax(out)
       return out

    #LSTM Configuration
batch_size = 3000
num_epochs = 20
learning_rate = 0.001#Check this learning rate

# Create LSTM
input_dim = 1       # input dimension
hidden_dim = 30     # hidden layer dimension
layer_dim = 15  # number of hidden layers
output_dim = 1      # output dimension
num_layers = 10      #num_layers

print("input_dim = ", input_dim,"\nhidden_dim = ", hidden_dim,"\nlayer_dim = ", layer_dim,"\noutput_dim = ", output_dim)

model = LSTMModel(input_dim, hidden_dim, layer_dim, output_dim)
model.cuda()
error = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

graph_index = 0
test_loss = []
train_loss = []
plt_test_index = []
plt_train_index = []
tmp_index = []
tmp_train = []
tmp_test = []

# model.init_hidden()

for epoch in range(num_epochs):

  # Train
  model.train()
  loss_list_train = []
  loss_list_test = []

  total_train = 0
  equals_train = 0
  total_test = 0
  num0_train = 0
  num1_train = 0
  num0_test = 0
  num1_test = 0
  equals_test = 0
  TP_train = 0
  FP_train = 0
  TN_train = 0
  FN_train = 0
  TP_test = 0
  FP_test = 0
  TN_test = 0
  FN_test = 0

#   for i, (inputs, targets) in enumerate(train_loader):
  for i, (inputs, targets) in enumerate(train_loader):
    train  = Variable(inputs.type(torch.FloatTensor).cuda())
    targets = Variable(targets.type(torch.FloatTensor).cuda())
    optimizer.zero_grad()
    outputs = model(train)
    loss = error(outputs, targets)
    loss_list_train.append(loss.item())
    loss.backward()
    # loss.backward(retain_graph=True)
    optimizer.step()
    t = np.where(targets.cpu().detach().numpy() > 0.5, 1, 0)
    o = np.where(outputs.cpu().detach().numpy() > 0.5, 1, 0)
    total_train += t.shape[0]
    equals_train += np.sum(t == o)
    num0_train += np.sum(t == 0)
    num1_train += np.sum(t == 1)
    TP_train += np.sum(np.logical_and(t == 1, o==1))
    FP_train += np.sum(np.logical_and(t == 1, o==0))
    TN_train += np.sum(np.logical_and(t == 0, o==0))
    FN_train += np.sum(np.logical_and(t == 0, o==1))
    tb.save_value('Train Loss', 'train_loss', globaliter, loss.item())
    globaliter += 1
    tb.flush_line('train_loss')
    print(i)

  # Test
  model.eval()
  targets_plot = np.array([])
  outputs_plot = np.array([])
  inputs_plot = np.array([])

  for inputs, targets in test_loader:

    inputs = Variable(inputs.type(torch.FloatTensor).cuda())
    targets = Variable(targets.type(torch.FloatTensor).cuda())
    outputs = model(inputs)
    loss = error(outputs, targets)
    loss_list_test.append(loss.item())
    #print(outputs.cpu().detach().numpy())
    t = np.where(targets.cpu().detach().numpy() > 0.5, 1, 0)
    o = np.where(outputs.cpu().detach().numpy() > 0.5, 1, 0)
    total_test += t.shape[0]
    equals_test += np.sum(t == o)
    num0_test += np.sum(t == 0)
    num1_test += np.sum(t == 1)
    TP_test += np.sum(np.logical_and(t == 1, o==1))
    FP_test += np.sum(np.logical_and(t == 0, o==1))
    TN_test += np.sum(np.logical_and(t == 0, o==0))
    FN_test += np.sum(np.logical_and(t == 1, o==0))
    tb.save_value('Test Loss', 'test_loss', globaliter2, loss.item())
    globaliter2 += 1
    tb.flush_line('test_loss')

  # Save value in array
  graph_index += 1
  plt_train_index.append(graph_index)
  plt_test_index.append(graph_index)
  train_loss.append(np.mean(np.array(loss_list_train)))
  test_loss.append(np.mean(np.array(loss_list_test)))

  print("------------------------------")
  print("Epoch : ", epoch)
  print("----- Train -----")
  print("Total =", total_train, " | Num 0 =", num0_train, " | Num 1 =", num1_train)
  print("Equals =", equals_train)
  print("Accuracy =", (equals_train / total_train)*100, "%")
  # print("TP =", TP_train / total_train, "% | TN =", TN_train / total_train, "% | FP =", FP_train / total_train, "% | FN =", FN_train / total_train, "%")
  print("----- Test -----")
  print("Total =", total_test, " | Num 0 =", num0_test, " | Num 1 =", num1_test)
  print("Equals =", equals_test)
  print("Accuracy =", (equals_test / total_test)*100, "%")

I am using the model to do binary classification on the sequence length of 300. The accuracy and the loss are not changing over several epochs.I tried changing the no. of layers,no of hidden states, activation function, but all to no avail. I don’t know what i am doing wrong, I am probably missing something fundamental. Any help is appreciated