title: Python to an exe File

body:
I am trying to convert my python file which uses transformers, spacy, and torch as packages, into an .exe file. I have tried both cx_freeze and pyinstaller but every time I try to run my generated exe I face the following errors respectively.

For Pyinstaller:

    C:\Users\aishv\output>qa.exe
To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html
Traceback (most recent call last):
  File "main.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\users\aishv\anaconda3\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\transformers\__init__.py", line 42, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\users\aishv\anaconda3\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\transformers\tokenization_auto.py", line 28, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\users\aishv\anaconda3\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\transformers\tokenization_xlm.py", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\users\aishv\anaconda3\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\sacremoses\__init__.py", line 2, in <module>
    """This directory is meant for IPython extensions."""
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "c:\users\aishv\anaconda3\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\sacremoses\tokenize.py", line 16, in <module>
  File "site-packages\sacremoses\tokenize.py", line 23, in MosesTokenizer
  File "site-packages\sacremoses\corpus.py", line 63, in chars
  File "pkgutil.py", line 637, in get_data
  File "c:\users\aishv\anaconda3\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 471, in get_data
    with open(path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\aishv\\AppData\\Local\\Temp\\_MEI228322\\sacremoses\\data\\perluniprops\\IsN.txt'
[23284] Failed to execute script main

For cx_freeze:

Traceback (most recent call last):
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\cx_Freeze\initscripts\__startup__.py", line 40, in run
    module.run()
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\cx_Freeze\initscripts\Console.py", line 37, in run
    exec(code, {'__name__': '__main__'})
  File "main.py", line 1, in <module>
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\transformers\__init__.py", line 20, in <module>
    from .file_utils import (TRANSFORMERS_CACHE, PYTORCH_TRANSFORMERS_CACHE, PYTORCH_PRETRAINED_BERT_CACHE,
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\transformers\file_utils.py", line 20, in <module>
    import boto3
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\boto3\__init__.py", line 16, in <module>
    from boto3.session import Session
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\boto3\session.py", line 17, in <module>
    import botocore.session
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\session.py", line 30, in <module>
    import botocore.client
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 16, in <module>
    from botocore import waiter, xform_name
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\waiter.py", line 18, in <module>
    from botocore.docs.docstring import WaiterDocstring
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\docs\__init__.py", line 15, in <module>
    from botocore.docs.service import ServiceDocumenter
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\docs\service.py", line 18, in <module>
    from botocore.docs.bcdoc.restdoc import DocumentStructure
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\docs\bcdoc\restdoc.py", line 16, in <module>
    from botocore.docs.bcdoc.docstringparser import DocStringParser
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\docs\bcdoc\docstringparser.py", line 16, in <module>
    class DocStringParser(six.moves.html_parser.HTMLParser):
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\vendored\six.py", line 92, in __get__
    result = self._resolve()
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\vendored\six.py", line 115, in _resolve
    return _import_module(self.mod)
  File "C:\Users\aishv\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\vendored\six.py", line 82, in _import_module
    __import__(name)
ModuleNotFoundError: No module named 'html.parser'

Please help me in this situation.
Here is my setup file for Cx_freeze:

from cx_Freeze import setup, Executable
setup(name='Qa',
        version='0.1',
        description='Application',
        executables = [Executable("main.py")])


title: get the index of element in NumPy array

body:
I have a Numpy integer array with a lot of duplicate elements.
For example:
a = np.random.randint(0,5,20)
a
Out[23]:
array([3, 1, 2, 4, 1, 2, 4, 3, 2, 3, 1, 4, 4, 1, 2, 4, 2, 4, 1, 1])

There are two cases:

if one element is less than 4, get all the indexes of this element
if one element is great than or equal to 4, select four of them randomly

I solved this with a loop.
ans = np.array([])
num = 4
for i in range(1,5):
    indexes = np.where(a == i)[0] # all indexes of elements equal to i
    index_i = np.random.choice(indexes, num, False) if len(indexes) >=num else indexes
    ans = np.concatenate([ans, index_i])

np.sort(ans)

Out[57]:
array([ 0.,  1.,  2.,  5.,  6.,  7.,  8.,  9., 10., 11., 13., 14., 15.,
       17., 19.])

Can I solve this problem without a loop or more efficiently in Numpy or PyTorch?


title: Hyperparameter optimization in pytorch (currently with sklearn GridSearchCV)

body:
I use this(link) pytorch tutorial and wish to add the grid search functionality in it ,sklearn.model_selection.GridSearchCV (link), in order to optimize the hyper parameters. I struggle in understanding what X and Y in gs.fit(x,y) should be; per the documentation (link) x and y are supposed to have the following structure but I have trouble figuring out how to get these off the code. The output of the class PennFudanDataset returns img and target in a form that does not align with the X, Y I need.
Are n_samples, n_features within the following block of code or in the tutorial’s block regarding the model?
fit(X, y=None, *, groups=None, **fit_params)[source]

Run fit with all sets of parameters.

Parameters
Xarray-like of shape (n_samples, n_features)
Training vector, where n_samples is the number of samples and n_features is the number of features.

yarray-like of shape (n_samples, n_output) or (n_samples,), default=None
Target relative to X for classification or regression; None for unsupervised learning.

Is there something else we could use instead that is easier to implement for this particular tutorial? I’ve read about ray tune(link), optuna(link) etc. but they seem more complex than that. I am currently also looking into scipy.optimize.brute(link) which seems simpler.
PennFundanDataset class:
import os
import numpy as np
import torch
from PIL import Image

class PennFudanDataset(object):
def __init__(self, root, transforms):
    self.root = root
    self.transforms = transforms
    # load all image files, sorting them to
    # ensure that they are aligned
    self.imgs = list(sorted(os.listdir(os.path.join(root, "PNGImages"))))
    self.masks = list(sorted(os.listdir(os.path.join(root, "PedMasks"))))

def __getitem__(self, idx):
    # load images ad masks
    img_path = os.path.join(self.root, "PNGImages", self.imgs[idx])
    mask_path = os.path.join(self.root, "PedMasks", self.masks[idx])
    img = Image.open(img_path).convert("RGB")
    # note that we haven't converted the mask to RGB,
    # because each color corresponds to a different instance
    # with 0 being background
    mask = Image.open(mask_path)
    # convert the PIL Image into a numpy array
    mask = np.array(mask)
    # instances are encoded as different colors
    obj_ids = np.unique(mask)
    # first id is the background, so remove it
    obj_ids = obj_ids[1:]

    # split the color-encoded mask into a set
    # of binary masks
    masks = mask == obj_ids[:, None, None]

    # get bounding box coordinates for each mask
    num_objs = len(obj_ids)
    boxes = []
    for i in range(num_objs):
        pos = np.where(masks[i])
        xmin = np.min(pos[1])
        xmax = np.max(pos[1])
        ymin = np.min(pos[0])
        ymax = np.max(pos[0])
        boxes.append([xmin, ymin, xmax, ymax])

    # convert everything into a torch.Tensor
    boxes = torch.as_tensor(boxes, dtype=torch.float32)
    # there is only one class
    labels = torch.ones((num_objs,), dtype=torch.int64)
    masks = torch.as_tensor(masks, dtype=torch.uint8)

    image_id = torch.tensor([idx])
    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    # suppose all instances are not crowd
    iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

    target = {}
    target["boxes"] = boxes
    target["labels"] = labels
    target["masks"] = masks
    target["image_id"] = image_id
    target["area"] = area
    target["iscrowd"] = iscrowd

    if self.transforms is not None:
        img, target = self.transforms(img, target)

    return img, target

def __len__(self):
    return len(self.imgs)


title: How do i add an layer to an Nural network in pytorch

body:
i want to add a layer to a Nural network programicaly it returned this error TypeError: forward() missing 1 required positional argument: 'x'

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.fc1 = nn.Linear(1, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

netz =Net()
print(netz)

netz = nn.Sequential([nn.Linear(10, 120), netz()])

print(netz)
`

The same error happend when i was loading it with netz=torch.load()

the line which seems to cause the error is :netz = nn.Sequential([nn.Linear(10, 120), netz()])

how do i make it work?


title: Multi-GPU training of AllenNLP coreference resolution

body:
I'm trying to replicate (or come close) to the results obtained by the End-to-end Neural Coreference Resolution paper on the CoNLL-2012 shared task. I intend to do some enhancements on top of this, so I decided to use AllenNLP's CoreferenceResolver. This is how I'm initialising & training the model:

import torch
from allennlp.common import Params
from allennlp.data import Vocabulary
from allennlp.data.dataset_readers import ConllCorefReader
from allennlp.data.dataset_readers.dataset_utils import Ontonotes
from allennlp.data.iterators import BasicIterator, MultiprocessIterator
from allennlp.data.token_indexers import SingleIdTokenIndexer, TokenCharactersIndexer
from allennlp.models import CoreferenceResolver
from allennlp.modules import Embedding, FeedForward
from allennlp.modules.seq2seq_encoders import PytorchSeq2SeqWrapper
from allennlp.modules.seq2vec_encoders import CnnEncoder
from allennlp.modules.text_field_embedders import BasicTextFieldEmbedder
from allennlp.modules.token_embedders import TokenCharactersEncoder
from allennlp.training import Trainer
from allennlp.training.learning_rate_schedulers import LearningRateScheduler
from torch.nn import LSTM, ReLU
from torch.optim import Adam

def read_data(directory_path):
    data = []
    for file_path in Ontonotes().dataset_path_iterator(directory_path):
        data += dataset_reader.read(file_path)
    return data

INPUT_FILE_PATH_TEMPLATE = "data/CoNLL-2012/v4/data/%s"
dataset_reader = ConllCorefReader(10, {"tokens": SingleIdTokenIndexer(),
                                       "token_characters": TokenCharactersIndexer()})
training_data = read_data(INPUT_FILE_PATH_TEMPLATE % "train")
validation_data = read_data(INPUT_FILE_PATH_TEMPLATE % "development")

vocabulary = Vocabulary.from_instances(training_data + validation_data)
model = CoreferenceResolver(vocab=vocabulary,
                            text_field_embedder=BasicTextFieldEmbedder({"tokens": Embedding.from_params(vocabulary, Params({"embedding_dim": embeddings_dimension, "pretrained_file": "glove.840B.300d.txt"})),
                                                                        "token_characters": TokenCharactersEncoder(embedding=Embedding(num_embeddings=vocabulary.get_vocab_size("token_characters"), embedding_dim=8, vocab_namespace="token_characters"),
                                                                                encoder=CnnEncoder(embedding_dim=8, num_filters=50, ngram_filter_sizes=(3, 4, 5), output_dim=100))}),
                            context_layer=PytorchSeq2SeqWrapper(LSTM(input_size=400, hidden_size=200, num_layers=1, dropout=0.2, bidirectional=True, batch_first=True)),
                            mention_feedforward=FeedForward(input_dim=1220, num_layers=2, hidden_dims=[150, 150], activations=[ReLU(), ReLU()], dropout=[0.2, 0.2]),
                            antecedent_feedforward=FeedForward(input_dim=3680, num_layers=2, hidden_dims=[150, 150], activations=[ReLU(), ReLU()], dropout=[0.2, 0.2]),
                            feature_size=20,
                            max_span_width=10,
                            spans_per_word=0.4,
                            max_antecedents=250,
                            lexical_dropout=0.5)

if torch.cuda.is_available():
    cuda_device = 0
    model = model.cuda(cuda_device)
else:
    cuda_device = -1

iterator = BasicIterator(batch_size=1)
iterator.index_with(vocabulary)
optimiser = Adam(model.parameters(), weight_decay=0.1)
Trainer(model=model,
        train_dataset=training_data,
        validation_dataset=validation_data,
        optimizer=optimiser,
        learning_rate_scheduler=LearningRateScheduler.from_params(optimiser, Params({"type": "step", "step_size": 100})),
        iterator=iterator,
        num_epochs=150,
        patience=1,
        cuda_device=cuda_device).train()

After reading the data I've trained the model but ran out of GPU memory: RuntimeError: CUDA out of memory. Tried to allocate 4.43 GiB (GPU 0; 11.17 GiB total capacity; 3.96 GiB already allocated; 3.40 GiB free; 3.47 GiB cached). Therefore, I attempted to make use of multiple GPUs to train this model. I'm making use of Tesla K80s (which have 12GiB memory).

I've tried making use of AllenNLP's MultiprocessIterator, by itialising the iterator as MultiprocessIterator(BasicIterator(batch_size=1), num_workers=torch.cuda.device_count()). However, only 1 GPU is being used (by monitoring the memory usage through the nvidia-smi command) & got the error below. I also tried fiddling with its parameters (increasing num_workers or decreasing output_queue_size) & the ulimit (as mentioned by this PyTorch issue) to no avail.

Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/allennlp/data/iterators/multiprocess_iterator.py", line 32, in _create_tensor_dicts
    output_queue.put(tensor_dict)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/allennlp/data/iterators/multiprocess_iterator.py", line 32, in _create_tensor_dicts
    output_queue.put(tensor_dict)
  File "<string>", line 2, in put
  File "<string>", line 2, in put
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client
    request = recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/home/user/.local/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 276, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
---------------------------------------------------------------------------

I also tried achieving this through PyTorch's DataParallel, by wrapping the model's context_layer, mention_feedforward, antecedent_feedforward with a custom DataParallelWrapper (to provide compatibility with the AllenNLP-assumed class functions). Still, only 1 GPU is used & it eventually runs out of memory as before.

class DataParallelWrapper(DataParallel):
    def __init__(self, module):
        super().__init__(module)

    def get_output_dim(self):
        return self.module.get_output_dim()

    def get_input_dim(self):
        return self.module.get_input_dim()

    def forward(self, *inputs):
        return self.module.forward(inputs)


title: Dynamic addition of hidden units in pytorch

body:
I am trying to add hidden units to a 3-layered neural network(input, hidden,output) dynamically as I train it. I want to keep the weights of trained part of the network as I add new hidden units.This is my code,

class my_network(torch.nn.Module):
    def __init__(self,input_dim,hidden_dim,output_dim):
        super(my_network,self).__init__()
        self.I = input_dim
        self.H = hidden_dim
        self.O = output_dim
        self.layer1 = torch.nn.Linear(input_dim,hidden_dim)
        self.layer2 = torch.nn.Linear(hidden_dim,output_dim)

    def add_neurons(self,no_of_neurons,flag):
        if flag == 1:
            weights = [self.layer1.weight.data,self.layer2.weight.data]
            self.layer1 = torch.nn.Linear(self.I,self.H+no_of_neurons)
            self.layer2 = torch.nn.Linear(self.H+no_of_neurons,self.O)
            self.layer1.weight.data[0:-no_of_neurons,:] = weights[0]
            self.layer2.weight.data[:,0:-no_of_neurons] = weights[1]
            self.H = self.H + no_of_neurons
        return self.layer1.weight.shape[0]

    def forward(self,x):
        temp = self.layer1(x)
        out = self.layer2(temp)
        return out

I have noticed that once I call “add_neurons” method, the weights stop updating(while gradients are generated). Any help would be much appreciated.


title: GAN, discriminator output only 0 or 1

body:
I'm trying to train SRGAN. (Super Resolution GAN)
However, the discriminator's output converge to 0 or 1 whatever the input is.
Discriminator's loss function is only

D_loss = 0.5*(D_net(fake) + 1 - D_net(real))

D_net(fake) and D_net(real) both becomes 0 or 1. (sigmoid)

How can I fix it?

    for epoch_idx in range(epoch_num):
      for batch_idx, data in enumerate(data_loader):
        D_net.zero_grad()
        #### make real, low, fake
        real = data[0]
        for img_idx in range(batch_size):
            low[img_idx] = trans_low_res(real[img_idx])
        fake = G_net(Variable(low).cuda())

        #### get Discriminator loss and train Discriminator
        real_D_out = D_net(Variable(real).cuda()).mean()
        fake_D_out = D_net(Variable(fake).cuda()).mean()

        D_loss = 0.5*(fake_D_out + 1 - real_D_out)
        D_loss.backward()
        D_optim.step()

        #### train Generator

        G_net.zero_grad()
        #### get new fake D out with updated Discriminator
        fake_D_out = D_net(Variable(fake).cuda()).mean()
        G_loss = generator_criterion(fake_D_out.cuda(), fake.cuda(), real.cuda())
        G_loss.backward()
        G_optim.step()

  Batch : [10/6700] Discriminator_Loss: 0.0860 Generator_Loss : 0.1393

  Batch : [20/6700] Discriminator_Loss: 0.0037 Generator_Loss : 0.1282

  Batch : [30/6700] Discriminator_Loss: 0.0009 Generator_Loss : 0.0838

  Batch : [40/6700] Discriminator_Loss: 0.0002 Generator_Loss : 0.0735

  Batch : [50/6700] Discriminator_Loss: 0.0001 Generator_Loss : 0.0648

  Batch : [60/6700] Discriminator_Loss: 0.5000 Generator_Loss : 0.0634

  Batch : [70/6700] Discriminator_Loss: 0.5000 Generator_Loss : 0.0706

  Batch : [80/6700] Discriminator_Loss: 0.5000 Generator_Loss : 0.0691

  Batch : [90/6700] Discriminator_Loss: 0.5000 Generator_Loss : 0.0538
  ...


title: One hot encoding a segmented image using pytorch

body:
I have a segmented image as a tensor of size [1,1,256,256]. The image is a binary segmented image. I want to one hot encode it to get an image of size [1,2,256,256].
I tried torch.nn.functional.one_hot(img, 2). But it gave me an image of size [1,256,256,2]. How do I get the desired tensor?


title: Using imbalanced-learn with Pandas DataFrame

body:
My dataset is quite imbalanced. The two minority classes each contain half of the sample in the majority class. My RNN model is not able to learn anything about the least populated class.

I'm trying to use the imbalanced-learn library. For instance:

sm = SMOTE(random_state=42, n_jobs=-1, k_neighbors=10)
X_train, y_train = sm.fit_resample(train.drop(['label], axis=1), train['label'])

works if train.drop(['label] contains just the values of the used features. The problem is that my DataFrame contains one additional columns containing strings as values: I cannot drop it since those strings are the input for my RNN. And if I drop it, I would not be able to tell to which row of the oversampled dataset those strings belong to.

Is there a way to keep all the columns and tell the function which columns to use for oversampling?


title: How to prepare this PyTorch official ImageNet example?

body:
This is a technical question on preparing a dataset.
I'm trying to follow this official example
https://github.com/pytorch/examples/tree/master/imagenet
but I cannot even start with because I don't understand the requirements. It says

Install PyTorch (pytorch.org)
pip install -r requirements.txt
Download the ImageNet dataset from http://www.image-net.org/
Then, and move validation images to labeled subfolders, using the following shell script

For the first requirement, I'm working on Colab, so I don't think I need to install PyTorch again on my local pc.
The second one doesn't work, as there's obviously no module named "requirements.txt". This is where I'm beginning to realize there's something on this git repo that I completely don't understand how to use. Anyway, I could just open the text file from the git repo directly, and it just says use torch and torchvision. Okay, I have no problem importing them.
The third requirement. So I went to ImageNet website and signed the agreement for the research use. Now the requirement tells me to download THE ImageNet data, but I see bunch of various options there (like by published years, purposes like for a competition, resolution, etc.). Which one is THE DATASET?
I'm new to PyTorch, and I think I'm missing some protocol about how the PyTorch dev community provides examples via this way...
Any help will be appreciated. Thank you.


title: Error parsing text-format caffe.NetParameter: caffe.LayerParameter has no field named "upsample_param"

body:
I tried to convert darknet to caffe (being inspired by this source code: https://github.com/ChenYingpeng/caffe-yolov3 that regards conversion between pytorch, darknet and caffe). However, I got this error :

[libprotobuf ERROR google/protobuf/text_format.cc:274] Error parsing text-format caffe.NetParameter: 2622:20: Message type "caffe.LayerParameter" has no field named "upsample_param".
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0224 23:43:05.536036 7907 upgrade_proto.cpp:90] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: yolov3.prototxt
*** Check failure stack trace: ***
Aborted (core dumped)

this is the code:

# The caffe module needs to be on the Python path;
#  we'll add it here explicitly.
caffe_root='/home/chen/caffe/'
#os.chdir(caffe_root)
import sys
sys.path.insert(0,caffe_root+'python')
import caffe
import numpy as np
from collections import OrderedDict
from cfg import *
from prototxt import *

def darknet2caffe(cfgfile, weightfile, protofile, caffemodel):
    net_info = cfg2prototxt(cfgfile)
    save_prototxt(net_info , protofile, region=False)

    net = caffe.Net(protofile, caffe.TEST)
    params = net.params

    blocks = parse_cfg(cfgfile)

    #Open the weights file
    fp = open(weightfile, "rb")

    #The first 4 values are header information
    # 1. Major version number
    # 2. Minor Version Number
    # 3. Subversion number
    # 4. IMages seen
    header = np.fromfile(fp, dtype = np.int32, count = 5)

    #fp = open(weightfile, 'rb')
    #header = np.fromfile(fp, count=5, dtype=np.int32)
    #header = np.ndarray(shape=(5,),dtype='int32',buffer=fp.read(20))
    #print(header)
    buf = np.fromfile(fp, dtype = np.float32)
    #print(buf)
    fp.close()

    layers = []
    layer_id = 1
    start = 0
    for block in blocks:
        if start >= buf.size:
            break

        if block['type'] == 'net':
            continue
        elif block['type'] == 'convolutional':
            batch_normalize = int(block['batch_normalize'])
            if block.has_key('name'):
                conv_layer_name = block['name']
                bn_layer_name = '%s-bn' % block['name']
                scale_layer_name = '%s-scale' % block['name']
            else:
                conv_layer_name = 'layer%d-conv' % layer_id
                bn_layer_name = 'layer%d-bn' % layer_id
                scale_layer_name = 'layer%d-scale' % layer_id

            if batch_normalize:
                start = load_conv_bn2caffe(buf, start, params[conv_layer_name], params[bn_layer_name], params[scale_layer_name])
            else:
                start = load_conv2caffe(buf, start, params[conv_layer_name])
            layer_id = layer_id+1
        elif block['type'] == 'depthwise_convolutional':
            batch_normalize = int(block['batch_normalize'])
            if block.has_key('name'):
                conv_layer_name = block['name']
                bn_layer_name = '%s-bn' % block['name']
                scale_layer_name = '%s-scale' % block['name']
            else:
                conv_layer_name = 'layer%d-dwconv' % layer_id
                bn_layer_name = 'layer%d-bn' % layer_id
                scale_layer_name = 'layer%d-scale' % layer_id

            if batch_normalize:
                start = load_conv_bn2caffe(buf, start, params[conv_layer_name], params[bn_layer_name], params[scale_layer_name])
            else:
                start = load_conv2caffe(buf, start, params[conv_layer_name])
            layer_id = layer_id+1
        elif block['type'] == 'connected':
            if block.has_key('name'):
                fc_layer_name = block['name']
            else:
                fc_layer_name = 'layer%d-fc' % layer_id
            start = load_fc2caffe(buf, start, params[fc_layer_name])
            layer_id = layer_id+1
        elif block['type'] == 'maxpool':
            layer_id = layer_id+1
        elif block['type'] == 'avgpool':
            layer_id = layer_id+1
        elif block['type'] == 'region':
            layer_id = layer_id + 1
        elif block['type'] == 'route':
            layer_id = layer_id + 1
        elif block['type'] == 'shortcut':
            layer_id = layer_id + 1
        elif block['type'] == 'softmax':
            layer_id = layer_id + 1
        elif block['type'] == 'cost':
            layer_id = layer_id + 1
    elif block['type'] == 'upsample':
        layer_id = layer_id + 1
        else:
            print('unknow layer type %s ' % block['type'])
            layer_id = layer_id + 1
    print('save prototxt to %s' % protofile)
    save_prototxt(net_info , protofile, region=True)
    print('save caffemodel to %s' % caffemodel)
    net.save(caffemodel)

def load_conv2caffe(buf, start, conv_param):
    weight = conv_param[0].data
    bias = conv_param[1].data
    conv_param[1].data[...] = np.reshape(buf[start:start+bias.size], bias.shape);   start = start + bias.size
    conv_param[0].data[...] = np.reshape(buf[start:start+weight.size], weight.shape); start = start + weight.size
    return start

def load_fc2caffe(buf, start, fc_param):
    weight = fc_param[0].data
    bias = fc_param[1].data
    fc_param[1].data[...] = np.reshape(buf[start:start+bias.size], bias.shape);   start = start + bias.size
    fc_param[0].data[...] = np.reshape(buf[start:start+weight.size], weight.shape); start = start + weight.size
    return start

def load_conv_bn2caffe(buf, start, conv_param, bn_param, scale_param):
    conv_weight = conv_param[0].data
    running_mean = bn_param[0].data
    running_var = bn_param[1].data
    scale_weight = scale_param[0].data
    scale_bias = scale_param[1].data

    scale_param[1].data[...] = np.reshape(buf[start:start+scale_bias.size], scale_bias.shape); start = start + scale_bias.size
    #print scale_bias.size
    #print scale_bias

    scale_param[0].data[...] = np.reshape(buf[start:start+scale_weight.size], scale_weight.shape); start = start + scale_weight.size
    #print scale_weight.size

    bn_param[0].data[...] = np.reshape(buf[start:start+running_mean.size], running_mean.shape); start = start + running_mean.size
    #print running_mean.size

    bn_param[1].data[...] = np.reshape(buf[start:start+running_var.size], running_var.shape); start = start + running_var.size
    #print running_var.size

    bn_param[2].data[...] = np.array([1.0])
    conv_param[0].data[...] = np.reshape(buf[start:start+conv_weight.size], conv_weight.shape); start = start + conv_weight.size
    #print conv_weight.size

    return start

def cfg2prototxt(cfgfile):
    blocks = parse_cfg(cfgfile)

    prev_filters = 3
    layers = []
    props = OrderedDict()
    bottom = 'data'
    layer_id = 1
    topnames = dict()
    for block in blocks:
        if block['type'] == 'net':
            props['name'] = 'Darkent2Caffe'
            props['input'] = 'data'
            props['input_dim'] = ['1']
            props['input_dim'].append(block['channels'])
            props['input_dim'].append(block['height'])
            props['input_dim'].append(block['width'])
            continue
        elif block['type'] == 'convolutional':
            conv_layer = OrderedDict()
            conv_layer['bottom'] = bottom
            if block.has_key('name'):
                conv_layer['top'] = block['name']
                conv_layer['name'] = block['name']
            else:
                conv_layer['top'] = 'layer%d-conv' % layer_id
                conv_layer['name'] = 'layer%d-conv' % layer_id
            conv_layer['type'] = 'Convolution'
            convolution_param = OrderedDict()
            convolution_param['num_output'] = block['filters']
            prev_filters = block['filters']
            convolution_param['kernel_size'] = block['size']
            if block['pad'] == '1':
                convolution_param['pad'] = str(int(convolution_param['kernel_size'])/2)
            convolution_param['stride'] = block['stride']
            if block['batch_normalize'] == '1':
                convolution_param['bias_term'] = 'false'
            else:
                convolution_param['bias_term'] = 'true'
            conv_layer['convolution_param'] = convolution_param
            layers.append(conv_layer)
            bottom = conv_layer['top']

            if block['batch_normalize'] == '1':
                bn_layer = OrderedDict()
                bn_layer['bottom'] = bottom
                bn_layer['top'] = bottom
                if block.has_key('name'):
                    bn_layer['name'] = '%s-bn' % block['name']
                else:
                    bn_layer['name'] = 'layer%d-bn' % layer_id
                bn_layer['type'] = 'BatchNorm'
                batch_norm_param = OrderedDict()
                batch_norm_param['use_global_stats'] = 'true'
                bn_layer['batch_norm_param'] = batch_norm_param
                layers.append(bn_layer)

                scale_layer = OrderedDict()
                scale_layer['bottom'] = bottom
                scale_layer['top'] = bottom
                if block.has_key('name'):
                    scale_layer['name'] = '%s-scale' % block['name']
                else:
                    scale_layer['name'] = 'layer%d-scale' % layer_id
                scale_layer['type'] = 'Scale'
                scale_param = OrderedDict()
                scale_param['bias_term'] = 'true'
                scale_layer['scale_param'] = scale_param
                layers.append(scale_layer)

            if block['activation'] != 'linear':
                relu_layer = OrderedDict()
                relu_layer['bottom'] = bottom
                relu_layer['top'] = bottom
                if block.has_key('name'):
                    relu_layer['name'] = '%s-act' % block['name']
                else:
                    relu_layer['name'] = 'layer%d-act' % layer_id
                relu_layer['type'] = 'ReLU'
                if block['activation'] == 'leaky':
                    relu_param = OrderedDict()
                    relu_param['negative_slope'] = '0.1'
                    relu_layer['relu_param'] = relu_param
                layers.append(relu_layer)
            topnames[layer_id] = bottom
            layer_id = layer_id+1
        elif block['type'] == 'depthwise_convolutional':
            conv_layer = OrderedDict()
            conv_layer['bottom'] = bottom
            if block.has_key('name'):
                conv_layer['top'] = block['name']
                conv_layer['name'] = block['name']
            else:
                conv_layer['top'] = 'layer%d-dwconv' % layer_id
                conv_layer['name'] = 'layer%d-dwconv' % layer_id
            conv_layer['type'] = 'ConvolutionDepthwise'
            convolution_param = OrderedDict()
            convolution_param['num_output'] = prev_filters
            convolution_param['kernel_size'] = block['size']
            if block['pad'] == '1':
                convolution_param['pad'] = str(int(convolution_param['kernel_size'])/2)
            convolution_param['stride'] = block['stride']
            if block['batch_normalize'] == '1':
                convolution_param['bias_term'] = 'false'
            else:
                convolution_param['bias_term'] = 'true'
            conv_layer['convolution_param'] = convolution_param
            layers.append(conv_layer)
            bottom = conv_layer['top']

            if block['batch_normalize'] == '1':
                bn_layer = OrderedDict()
                bn_layer['bottom'] = bottom
                bn_layer['top'] = bottom
                if block.has_key('name'):
                    bn_layer['name'] = '%s-bn' % block['name']
                else:
                    bn_layer['name'] = 'layer%d-bn' % layer_id
                bn_layer['type'] = 'BatchNorm'
                batch_norm_param = OrderedDict()
                batch_norm_param['use_global_stats'] = 'true'
                bn_layer['batch_norm_param'] = batch_norm_param
                layers.append(bn_layer)

                scale_layer = OrderedDict()
                scale_layer['bottom'] = bottom
                scale_layer['top'] = bottom
                if block.has_key('name'):
                    scale_layer['name'] = '%s-scale' % block['name']
                else:
                    scale_layer['name'] = 'layer%d-scale' % layer_id
                scale_layer['type'] = 'Scale'
                scale_param = OrderedDict()
                scale_param['bias_term'] = 'true'
                scale_layer['scale_param'] = scale_param
                layers.append(scale_layer)

            if block['activation'] != 'linear':
                relu_layer = OrderedDict()
                relu_layer['bottom'] = bottom
                relu_layer['top'] = bottom
                if block.has_key('name'):
                    relu_layer['name'] = '%s-act' % block['name']
                else:
                    relu_layer['name'] = 'layer%d-act' % layer_id
                relu_layer['type'] = 'ReLU'
                if block['activation'] == 'leaky':
                    relu_param = OrderedDict()
                    relu_param['negative_slope'] = '0.1'
                    relu_layer['relu_param'] = relu_param
                layers.append(relu_layer)
            topnames[layer_id] = bottom
            layer_id = layer_id+1
        elif block['type'] == 'maxpool':
            max_layer = OrderedDict()
            max_layer['bottom'] = bottom
            if block.has_key('name'):
                max_layer['top'] = block['name']
                max_layer['name'] = block['name']
            else:
                max_layer['top'] = 'layer%d-maxpool' % layer_id
                max_layer['name'] = 'layer%d-maxpool' % layer_id
            max_layer['type'] = 'Pooling'
            pooling_param = OrderedDict()
            pooling_param['stride'] = block['stride']
            pooling_param['pool'] = 'MAX'
            if (int(block['size']) - int(block['stride'])) % 2 == 0:
        pooling_param['kernel_size'] = block['size']
                pooling_param['pad'] = str((int(block['size'])-1)/2)

            if (int(block['size']) - int(block['stride'])) % 2 == 1:
                pooling_param['kernel_size'] = str(int(block['size']) + 1)
                pooling_param['pad'] = str((int(block['size']) + 1)/2)

            max_layer['pooling_param'] = pooling_param
            layers.append(max_layer)
            bottom = max_layer['top']
            topnames[layer_id] = bottom
            layer_id = layer_id+1
        elif block['type'] == 'avgpool':
            avg_layer = OrderedDict()
            avg_layer['bottom'] = bottom
            if block.has_key('name'):
                avg_layer['top'] = block['name']
                avg_layer['name'] = block['name']
            else:
                avg_layer['top'] = 'layer%d-avgpool' % layer_id
                avg_layer['name'] = 'layer%d-avgpool' % layer_id
            avg_layer['type'] = 'Pooling'
            pooling_param = OrderedDict()
            pooling_param['kernel_size'] = 7
            pooling_param['stride'] = 1
            pooling_param['pool'] = 'AVE'
            avg_layer['pooling_param'] = pooling_param
            layers.append(avg_layer)
            bottom = avg_layer['top']
            topnames[layer_id] = bottom
            layer_id = layer_id+1
        elif block['type'] == 'region':
            if True:
                region_layer = OrderedDict()
                region_layer['bottom'] = bottom
                if block.has_key('name'):
                    region_layer['top'] = block['name']
                    region_layer['name'] = block['name']
                else:
                    region_layer['top'] = 'layer%d-region' % layer_id
                    region_layer['name'] = 'layer%d-region' % layer_id
                region_layer['type'] = 'Region'
                region_param = OrderedDict()
                region_param['anchors'] = block['anchors'].strip()
                region_param['classes'] = block['classes']
                region_param['num'] = block['num']
                region_layer['region_param'] = region_param
                layers.append(region_layer)
                bottom = region_layer['top']
            topnames[layer_id] = bottom
            layer_id = layer_id + 1

        elif block['type'] == 'route':
            route_layer = OrderedDict()
        layer_name = str(block['layers']).split(',')
        #print(layer_name[0])
        bottom_layer_size = len(str(block['layers']).split(','))
        #print(bottom_layer_size)
        if(1 == bottom_layer_size):
                prev_layer_id = layer_id + int(block['layers'])
                bottom = topnames[prev_layer_id]
                #topnames[layer_id] = bottom
        route_layer['bottom'] = bottom
        if(2 == bottom_layer_size):
        prev_layer_id1 = layer_id + int(layer_name[0])
        #print(prev_layer_id1)
        prev_layer_id2 = int(layer_name[1]) + 1
        print(topnames)
        bottom1 = topnames[prev_layer_id1]
        bottom2 = topnames[prev_layer_id2]
        route_layer['bottom'] = [bottom1, bottom2]
        if(4 == bottom_layer_size):
        prev_layer_id1 = layer_id + int(layer_name[0])
        prev_layer_id2 = layer_id + int(layer_name[1])
        prev_layer_id3 = layer_id + int(layer_name[2])
        prev_layer_id4 = layer_id + int(layer_name[3])

        bottom1 = topnames[prev_layer_id1]
        bottom2 = topnames[prev_layer_id2]
        bottom3 = topnames[prev_layer_id3]
        bottom4 = topnames[prev_layer_id4]
        route_layer['bottom'] = [bottom1, bottom2,bottom3,bottom4]
        if block.has_key('name'):
                route_layer['top'] = block['name']
                route_layer['name'] = block['name']
            else:
                route_layer['top'] = 'layer%d-route' % layer_id
                route_layer['name'] = 'layer%d-route' % layer_id
        route_layer['type'] = 'Concat'
        print(route_layer)
        layers.append(route_layer)
        bottom = route_layer['top']
        print(layer_id)
            topnames[layer_id] = bottom
        layer_id = layer_id + 1

    elif block['type'] == 'upsample':
        upsample_layer = OrderedDict()
        print(block['stride'])
        upsample_layer['bottom'] = bottom
        if block.has_key('name'):
                upsample_layer['top'] = block['name']
                upsample_layer['name'] = block['name']
            else:
                upsample_layer['top'] = 'layer%d-upsample' % layer_id
                upsample_layer['name'] = 'layer%d-upsample' % layer_id
        upsample_layer['type'] = 'Upsample'
        upsample_param = OrderedDict()
        upsample_param['scale'] = block['stride']
        upsample_layer['upsample_param'] = upsample_param
        print(upsample_layer)
        layers.append(upsample_layer)
        bottom = upsample_layer['top']
        print('upsample:',layer_id)
            topnames[layer_id] = bottom
        layer_id = layer_id + 1

        elif block['type'] == 'shortcut':
            prev_layer_id1 = layer_id + int(block['from'])
            prev_layer_id2 = layer_id - 1
            bottom1 = topnames[prev_layer_id1]
            bottom2= topnames[prev_layer_id2]
            shortcut_layer = OrderedDict()
            shortcut_layer['bottom'] = [bottom1, bottom2]
            if block.has_key('name'):
                shortcut_layer['top'] = block['name']
                shortcut_layer['name'] = block['name']
            else:
                shortcut_layer['top'] = 'layer%d-shortcut' % layer_id
                shortcut_layer['name'] = 'layer%d-shortcut' % layer_id
            shortcut_layer['type'] = 'Eltwise'
            eltwise_param = OrderedDict()
            eltwise_param['operation'] = 'SUM'
            shortcut_layer['eltwise_param'] = eltwise_param
            layers.append(shortcut_layer)
            bottom = shortcut_layer['top']

            if block['activation'] != 'linear':
                relu_layer = OrderedDict()
                relu_layer['bottom'] = bottom
                relu_layer['top'] = bottom
                if block.has_key('name'):
                    relu_layer['name'] = '%s-act' % block['name']
                else:
                    relu_layer['name'] = 'layer%d-act' % layer_id
                relu_layer['type'] = 'ReLU'
                if block['activation'] == 'leaky':
                    relu_param = OrderedDict()
                    relu_param['negative_slope'] = '0.1'
                    relu_layer['relu_param'] = relu_param
                layers.append(relu_layer)
            topnames[layer_id] = bottom
            layer_id = layer_id + 1

        elif block['type'] == 'connected':
            fc_layer = OrderedDict()
            fc_layer['bottom'] = bottom
            if block.has_key('name'):
                fc_layer['top'] = block['name']
                fc_layer['name'] = block['name']
            else:
                fc_layer['top'] = 'layer%d-fc' % layer_id
                fc_layer['name'] = 'layer%d-fc' % layer_id
            fc_layer['type'] = 'InnerProduct'
            fc_param = OrderedDict()
            fc_param['num_output'] = int(block['output'])
            fc_layer['inner_product_param'] = fc_param
            layers.append(fc_layer)
            bottom = fc_layer['top']

            if block['activation'] != 'linear':
                relu_layer = OrderedDict()
                relu_layer['bottom'] = bottom
                relu_layer['top'] = bottom
                if block.has_key('name'):
                    relu_layer['name'] = '%s-act' % block['name']
                else:
                    relu_layer['name'] = 'layer%d-act' % layer_id
                relu_layer['type'] = 'ReLU'
                if block['activation'] == 'leaky':
                    relu_param = OrderedDict()
                    relu_param['negative_slope'] = '0.1'
                    relu_layer['relu_param'] = relu_param
                layers.append(relu_layer)
            topnames[layer_id] = bottom
            layer_id = layer_id+1
        else:
            print('unknow layer type %s ' % block['type'])
            topnames[layer_id] = bottom
            layer_id = layer_id + 1

    net_info = OrderedDict()
    net_info['props'] = props
    net_info['layers'] = layers
    return net_info

if __name__ == '__main__':
    import sys
    if len(sys.argv) != 5:
        print('try:')
        print('python darknet2caffe.py tiny-yolo-voc.cfg tiny-yolo-voc.weights tiny-yolo-voc.prototxt tiny-yolo-voc.caffemodel')
        print('')
        print('please add name field for each block to avoid generated name')
        exit()

    cfgfile = sys.argv[1]
    #net_info = cfg2prototxt(cfgfile)
    #print_prototxt(net_info)
    #save_prototxt(net_info, 'tmp.prototxt')
    weightfile = sys.argv[2]
    protofile = sys.argv[3]
    caffemodel = sys.argv[4]
    darknet2caffe(cfgfile, weightfile, protofile, caffemodel)

I followed all steps described here, I even created the upsample layer (by creating three files and editing caffe.proto.
I am wondering why this error occurs since I created upsampling layer.
any suggestion will be helpful.


title: Output dimension of a custom LSTM model in Pytorch

body:
I have a custom LSTM model in PyTorch like below:

hidden_size = 32
num_layers = 1
num_classes = 2

class customModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(customModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bilstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional=True)
        self.fcl = nn.Linear(hidden_size*2, num_classes)

    def forward(self, x):
        # Set initial hidden and cell states
        h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device)
        c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device)

        # Forward propagate LSTM
        out, hidden = self.bilstm(x, (h0, c0))
        fw_bilstm = out[-1, :, :self.hidden_size]
        bk_bilstm = out[0, :, :self.hidden_size]
        concat_fw_bw = torch.cat((fw_bilstm, bk_bilstm), dim = 1)
        fc = self.fcl(concat_fw_bw)
        x = F.softmax(F.relu(fc))
        return x

I can pass an input of type torch.Tensor to this model. The input is of length 67349, each is a 300 dimension vector.

After model initiation and prediction, I get an output vector of length 1.

model = customModel(300, hidden_size, num_layers, num_classes)
output = model(input_torch)

The output shows tensor([[0.5020, 0.4980]], grad_fn=<SoftmaxBackward>) when I print it out.

Why is this output of length 1? It seems like I should NOT have barch_first=True in my model but changing that requires other input dimension changes which I am not sure how to do.

Please suggest how can I get a vector output of length 67349 (input length) instead of 1?

Explanation

I see @gorjan suggested some modifications in the forward method of the network. So I wanted to clarify more what I am trying to build

Feed the embedding to a BiLSTM (done)
Get the hidden states of the last step in each direction and concatenate
Fed the concatenated output (from step 2) to a fully connected layer
with ReLUs
Fed the output from step 3 to a softmax layer


title: In the PyTorch C++ extension, how can I access a single element in a tensor and convert it to a standard c++ datatype?

body:
I am writing a c++ extension for pytorch, in which I need to access the elements of a tensor by index, and I also need to convert the element to a standard c++ type. Here is a short example. Suppose I have a 2d tensor a and I need to access a[i][j] and convert it to float.
#include <torch/extension.h>

float get(torch::Tensor a, int i, int j) {
    return a[i][j];
}

The above is put into a file called tensortest.cpp. In another file setup.py I write
from setuptools import setup, Extension
from torch.utils import cpp_extension

setup(name='tensortest',
      ext_modules=[cpp_extension.CppExtension('tensortest_cpp', ['tensortest.cpp'])],
      cmdclass={'build_ext': cpp_extension.BuildExtension})

When I run python setup.py install the compiler reports the following error
running install
running bdist_egg
running egg_info
creating tensortest.egg-info
writing tensortest.egg-info/PKG-INFO
writing dependency_links to tensortest.egg-info/dependency_links.txt
writing top-level names to tensortest.egg-info/top_level.txt
writing manifest file 'tensortest.egg-info/SOURCES.txt'
/home/trisst/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py:335: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'tensortest.egg-info/SOURCES.txt'
writing manifest file 'tensortest.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'tensortest_cpp' extension
creating build
creating build/temp.linux-x86_64-3.8
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/user/.local/lib/python3.8/site-packages/torch/include -I/home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/user/.local/lib/python3.8/site-packages/torch/include/TH -I/home/user/.local/lib/python3.8/site-packages/torch/include/THC -I/usr/include/python3.8 -c tensortest.cpp -o build/temp.linux-x86_64-3.8/tensortest.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=tensortest_cpp -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
In file included from /home/user/.local/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:149,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
                 from tensortest.cpp:1:
/home/user/.local/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:84: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
   84 | #pragma omp parallel for if ((end - begin) >= grain_size)
      |
tensortest.cpp: In function ‘float get(at::Tensor, int, int)’:
tensortest.cpp:4:15: error: cannot convert ‘at::Tensor’ to ‘float’ in return
    4 |  return a[i][j];
      |               ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

What can I do?


title: Understanding ELMo's number of presentations

body:
I am trying my hand at ELMo by simply using it as part of a larger PyTorch model. A basic example is given here.

  This is a torch.nn.Module subclass that computes any number of ELMo
  representations and introduces trainable scalar weights for each. For
  example, this code snippet computes two layers of representations (as
  in the SNLI and SQuAD models from our paper):

from allennlp.modules.elmo import Elmo, batch_to_ids

options_file = "https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_options.json"
weight_file = "https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5"

# Compute two different representation for each token.
# Each representation is a linear weighted combination for the
# 3 layers in ELMo (i.e., charcnn, the outputs of the two BiLSTM))
elmo = Elmo(options_file, weight_file, 2, dropout=0)

# use batch_to_ids to convert sentences to character ids
sentences = [['First', 'sentence', '.'], ['Another', '.']]
character_ids = batch_to_ids(sentences)

embeddings = elmo(character_ids)

# embeddings['elmo_representations'] is length two list of tensors.
# Each element contains one layer of ELMo representations with shape
# (2, 3, 1024).
#   2    - the batch size
#   3    - the sequence length of the batch
#   1024 - the length of each ELMo vector

My question concerns the 'representations'. Can you compare them to normal word2vec output layers? You can choose how many ELMo will give back (increasing an n-th dimension), but what is the difference between these generated representations and what is their typical use?

To give you an idea, for the above code, embeddings['elmo_representations'] returns a list of two items (the two representation layers) but they are identical.

In short, how can one define the 'representations' in ELMo?


title: How to properly convert pytorch LSTM to keras CuDNNLSTM?

body:
I am trying to hand-convert a Pytorch model to Tensorflow for deployment. ONNX doesn't seem to natively go from Pytorch LSTMs to Tensorflow CuDNNLSTMs so that's why I'm writing it by hand.

I've tried the code below:
This is running in an Anaconda environment running Python 2.7, Pytorch 1.0, tensorflow 1.12, cuda9. I'm running this with no bias in the Pytorch layer as it follows a batchnorm, but since Keras does not provide that option I'm simply assigning a 0 bias.

import torch
import tensorflow as tf
import numpy as np
from tensorflow.keras.layers import CuDNNLSTM, Bidirectional
from tensorflow.keras.models import Sequential, Model

input_size = 80
hidden_size = 512
with torch.no_grad():
    rnn1 = torch.nn.LSTM(input_size=input_size, hidden_size=hidden_size, bidirectional=True, bias=False, batch_first=True).cuda()

model = Sequential()
model.add(Bidirectional(CuDNNLSTM(hidden_size, return_sequences=True),  input_shape=(None, input_size), name='rnn'))

bias_size = rnn1.weight_hh_l0.detach().cpu().numpy().T.shape[1] * 2
keras_format_weights = [
                    rnn1.weight_ih_l0.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0.detach().cpu().numpy().T,
                    np.zeros(bias_size,),
                    rnn1.weight_ih_l0_reverse.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0_reverse.detach().cpu().numpy().T,
                    np.zeros(bias_size,),
                  ]

model.layers[0].set_weights(keras_format_weights)

random_test = np.random.rand(1, 1, 80)

res1, _ = rnn1.forward(torch.FloatTensor(random_test).cuda())
res1 = res1.detach().cpu().numpy()
res2 = model.predict(random_test)

print(np.allclose(res1, res2, atol=1e-2))
print(res1)
print(res2)

False
[[[ 0.01265562  0.07478553  0.0470101  ... -0.02260824  0.0243004
   -0.0261014 ]]]
[[[-0.05316251 -0.00230848  0.03070898 ...  0.01497027  0.00976444
   -0.01095549]]]

Now, this does work with the generic Keras LSTM:

model = Sequential()
model.add(Bidirectional(LSTM(hidden_size, recurrent_activation='sigmoid', return_sequences=True),  input_shape=(None, input_size), name='rnn'))

bias_size = rnn1.weight_hh_l0.detach().cpu().numpy().T.shape[1]
keras_format_weights = [
                    rnn1.weight_ih_l0.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0.detach().cpu().numpy().T,
                    np.zeros((bias_size,)),
                    rnn1.weight_ih_l0_reverse.detach().cpu().numpy().T,
                    rnn1.weight_hh_l0_reverse.detach().cpu().numpy().T,
                    np.zeros((bias_size,))
                  ]

But I need the speed advantages of the CuDNNLSTM, and Pytorch is using the same backend anyway.


title: Pytorch Build from Source gives Error make: *** No rule to make target 'install'. Stop

body:
I am following this guide to build Pytorch from scratch on a Raspberry Pi3B. For some reason, there is an error:

Building wheel torch-1.2.0a0+f13fadd
-- Building version 1.2.0a0+f13fadd
cmake --build . --target install --config Release -- -j 4
make: *** No rule to make target 'install'.  Stop.

when I call python3 setup.py build. I am running Python version 3.5 and I am unsure why this seems to be failing.


title: Apply json config file using parse_args() in pycharm

body:
I'm running sequence to sequence code in git, but I got error about parse_args().
my code is like this:

parser = argparse.ArgumentParser()
parser.add_argument(
    "--config",
    help="path to json config",
    required=True)
args = parser.parse_args()
config_file_path = args.config
config = read_config(config_file_path)
experiment_name = hyperparam_string(config)

my config file is like this :

{
  "training": {
    "optimizer": "adam",
    "clip_c": 1,
    "lrate": 0.0002,
  },
  "management": {
    "monitor_loss": 1000,
    "print_samples": 20000
  }

When I run

args = parser.parse_args()

Pycharm raises error

pydevconsole.py: error: argument --config is required

I'd like to know that run this code through pycharm applying json config file.

I have searched in google since yesterday, but I can't find it.

Please help..


title: Using conda to install PyTorch within a Visual Studio python project

body:
I've created a Python project within Visual Studio 2017 Community Edition.

I've changed the environment within the Visual Studio Python project to be Anaconda 5.2.

If I just use the "Install Python Package..." menu in Visual Studio and type PyTorch, it's says the package is not found.

So, I thought next I should just run this command...

conda install pytorch -c pytorch

If I open the Python Environments within Visual Studio, click on Anaconda 5.2, and click Open in Powershell, and execute the command above in the Powershell terminal, it eventually fails because of a permission error.  Guessing this is due to not having Administrative Privileges.

If I just open the Run menu in Windows and open a Powershell with Administrative Privileges then "conda" is not found typing the command above.

How should I be installing PyTorch in the Visual Studio Anaconda environment?


title: How to give a batch of frames to the model in pytorch c++ api?

body:
I've written a code to load the pytorch model in C++ with help of the PyTorch C++ Frontend api. I want to give a batch of frames to a pretrained model in the C++ by using module->forward(batch_frames). But it can forward through a single input.
How can I give a batch of inputs to the model?

A part of code that I want to give the batch is shown below:

 cv::Mat frame;
 vector<torch::jit::IValue> frame_batch;

 // do some pre-processes on each frame and then add it to the frame_batch

 //forward through the batch frames
 torch::Tensor output = module->forward(frame_batch).toTensor();


title: What is projection layer in the context of neural machine translation using RNN?

body:
I read a paper about machine translation, and it uses projection layer.
The projection layer is explained as follows: "Additional projection aims to reduce the dimensionality of the encoder output representations to match the decoder stack dimension."

Does anyone know this architecture or how to implement this layer in Pytorch?

The paper's link: https://www.aclweb.org/anthology/P18-1008.pdf

The model architecture:


title: Pytorch "upsample_bilinear2d_out_frame" not implemented for 'Byte'

body:
I have trained a custom object detection model using the steps described in this link. I am able to train my model but when I try to evaluate it at the end of an epoch, I get the following error

Epoch: [0] Total time: 0:00:06 (0.2223 s / it)
creating index...
index created!
Traceback (most recent call last):
  File "train.py", line 106, in <module>
    evaluate(model, data_loader_test, device=device)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/home/sarvani/Desktop/flir/test_frcnn/custom/engine.py", line 107, in evaluate
    outputs = model(image)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 47, in forward
    images, targets = self.transform(images, targets)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torchvision/models/detection/transform.py", line 41, in forward
    image, target = self.resize(image, target)
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torchvision/models/detection/transform.py", line 70, in resize
    image[None], scale_factor=scale_factor, mode='bilinear', align_corners=False)[0]
  File "/home/sarvani/anaconda3/envs/flir_env/lib/python3.7/site-packages/torch/nn/functional.py", line 2503, in interpolate
    return torch._C._nn.upsample_bilinear2d(input, _output_size(2), align_corners)
RuntimeError: "upsample_bilinear2d_out_frame" not implemented for 'Byte'

My code to load the data is as follows

class CustomDataset(torch.utils.data.Dataset):

    def __init__(self, root_dir,transform=None):
        self.root = root_dir
        self.rgb_imgs = list(sorted(os.listdir(os.path.join(root_dir, "rgb/"))))
        self.annotations = list(sorted(os.listdir(os.path.join(root_dir, "annotations/"))))

        self._classes = ('__background__',  # always index 0
                         'car','person','bicycle','dog','other')

        self._class_to_ind = {'car':'1', 'person':'2', 'bicycle':'3', 'dog':'4','other':'5'}

    def __len__(self):
        return len(self.rgb_imgs)

    def __getitem__(self, idx):
        self.num_classes = 6

        img_rgb_path = os.path.join(self.root, "rgb/", self.rgb_imgs[idx])
        img = Image.open(img_rgb_path)
        img = np.array(img)
        img = img.transpose((2, 0, 1))
        img = torch.from_numpy(img)

        filename = os.path.join(self.root,'annotations',self.annotations[idx])
        tree = ET.parse(filename)
        objs = tree.findall('object')

        num_objs = len(objs)
        labels = np.zeros((num_objs), dtype=np.float32)
        seg_areas = np.zeros((num_objs), dtype=np.float32)

        boxes = []
        for ix, obj in enumerate(objs):
            bbox = obj.find('bndbox')
            x1 = float(bbox.find('xmin').text)
            y1 = float(bbox.find('ymin').text)
            x2 = float(bbox.find('xmax').text)
            y2 = float(bbox.find('ymax').text)

            cls = self._class_to_ind[obj.find('name').text.lower().strip()]
            boxes.append([x1, y1, x2, y2])
            labels[ix] = cls

        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        image_id = torch.tensor([idx])
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
        labels = torch.as_tensor(labels, dtype=torch.float32)

        target =  {'boxes': boxes,
                'labels': labels,
                'area': area,
                "image_id":image_id
                }
        target["iscrowd"] = iscrowd
        return img,target

My train.py is as follows

num_classes = 6
model = fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

device = torch.device('cuda')
model = model.cuda()

dataset_train = CustomDataset('FLIR/images/train')
dataset_val = CustomDataset('FLIR/images/val')

data_loader_train = torch.utils.data.DataLoader(
    dataset_train, batch_size=4, shuffle=True,collate_fn=utils.collate_fn)

data_loader_test = torch.utils.data.DataLoader(
    dataset_val, batch_size=4 shuffle=False,collate_fn=utils.collate_fn)

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.Adam(params,lr=0.05,weight_decay=0.0005)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=3,
                                               gamma=0.1)

num_epochs = 30

for epoch in range(num_epochs):
    train_one_epoch(model, optimizer, data_loader_train, device, epoch, print_freq=1)
    lr_scheduler.step()
    evaluate(model, data_loader_test, device=device)

The evaluation function long with the necessary files used are at this link.

Can someone please help me out.


title: Pytorch installation on windows

body:
I have tried many suggestions online to install in my virtual env "torch" but to no avail. It won't let me import torch. I am able to install torchvision through conda though following this link: https://pytorch.org/get-started/locally/. Any suggestions are welcome!

Here is the error message (I downgrade to python 3.5 in the virtualenv)

(env_peem) PS E:\Users\Maggie\TS_MatSeg_Share> pip install torch
Collecting torch\Users\Maggie\TS_MatSeg_Share>
  Using cached torch-0.1.2.post2.tar.gz (128 kB)
Requirement already satisfied: pyyaml in e:\users\maggie\ts_matseg_share\env_peem\lib\site-packages (from torch) (5.3.1)
Building wheels for collected packages: torch
  Building wheel for torch (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"'; __file__='"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'E:\Users\Maggie\AppData\Local\Temp\7\pip-wheel-xn9ld0bu'
       cwd: E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\
  Complete output (30 lines):
  running bdist_wheel
  running build
  running build_deps
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 265, in <module>
      description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
    File "E:\Users\Maggie\TS_MatSeg_Share\env_peem\lib\site-packages\setuptools\__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "E:\Users\Maggie\TS_MatSeg_Share\env_peem\lib\site-packages\wheel\bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 51, in run
      from tools.nnwrap import generate_wrappers as generate_nn_wrappers
  ImportError: No module named 'tools.nnwrap'
  ----------------------------------------
  ERROR: Failed building wheel for torch
  Running setup.py clean for torch
  ERROR: Command errored out with exit status 1:
   command: 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"'; __file__='"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch
  Complete output (2 lines):
  running clean
  error: [Errno 2] No such file or directory: '.gitignore'
  ----------------------------------------
  ERROR: Failed cleaning build dir for torch
Failed to build torch
Installing collected packages: torch
    Running setup.py install for torch ... error
    ERROR: Command errored out with exit status 1:
     command: 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"'; __file__='"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'E:\Users\Maggie\AppData\Local\Temp\7\pip-record-z6gt4ig9\install-record.txt' --single-version-externally-managed --compile --install-headers 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\include\site\python3.5\torch'
         cwd: E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\
    Complete output (23 lines):
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "E:\Users\Maggie\TS_MatSeg_Share\env_peem\lib\site-packages\setuptools\__init__.py", line 144, in setup
        return distutils.core.setup(**attrs)
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 99, in run
        self.run_command('build_deps')
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\Maggie\AppData\Local\Programs\Python\Python35\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Users\Maggie\AppData\Local\Temp\7\pip-install-80ts7bt9\torch\setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ImportError: No module named 'tools.nnwrap'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"'; __file__='"'"'E:\\Users\\Maggie\\AppData\\Local\\Temp\\7\\pip-install-80ts7bt9\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'E:\Users\Maggie\AppData\Local\Temp\7\pip-record-z6gt4ig9\install-record.txt' --single-version-externally-managed --compile --install-headers 'E:\Users\Maggie\TS_MatSeg_Share\env_peem\include\site\python3.5\torch' Check the logs for full command output.


title: Broadcasting element wise multiplication in pytorch

body:
I have a tensor in pytorch with size torch.Size([1443747, 128]). Let's name it tensor A. In this tensor, 128 represents a batch size. I have another 1D tensor with size torch.Size([1443747]). Let's call it B. I want to do element wise multiplication of B with A, such that B is multiplied with all 128 columns of tensor A (obviously in an element wise manner). In other words, I want to broadcast the element wise multiplication along dimension=1.
How can I achieve this in pytorch?
It I didn't have a batch size involved in the tensor A (batch size = 1), then normal * operator would do the multiplication easily. A*B then would have generated resultant tensor of size torch.Size([1443747]). However, I don't understand why pytorch is not broadcasting the tensor multiplication along dimension 1? Is there any way to do this?
What I want is, B should be multiplied with all 128 columns of A in an element wise manner. So, the resultant tensors' size would be torch.Size([1443747, 128]).


title: Pytorch: What happens to memory when moving tensor to GPU?

body:
I’m trying to understand what happens to both RAM and GPU memory when a tensor is sent to the GPU.
In the following code sample, I create two tensors - large tensor arr = torch.Tensor.ones((10000, 10000)) and small tensor c = torch.Tensor.ones(1). Tensor c is sent to GPU inside the target function step which is called by multiprocessing.Pool. In doing so, each child process uses 487 MB on the GPU and RAM usage goes to 5 GB. Note that the large tensor arr is just created once before calling Pool and not passed as an argument to the target function. Ram usage does not explode when everything is on the CPU.
I have the following questions on this example:
I’m sending torch.Tensor.ones(1) to GPU and yet it consumes 487 MB of GPU memory. Does CUDA allocate a minimum amount of memory on the GPU even if the underlying tensor is very small? GPU memory is not a problem for me, and this is just for me to understand how the allocation is done.
The problem lies in RAM usage. Even though I am sending a small tensor to the GPU, it appears as if everything in memory (large tensor arr) is copied for every child process (possibly to pinned memory). So when a tensor is sent to the GPU, what objects are copied to pinned memory? I’m missing something here as it does not make sense to prepare everything to be sent to GPU when I’m only sending a particular object.
Thanks!
from multiprocessing import get_context
import time
import torch

dim = 10000
sleep_time = 2
npe = 4  # number of parallel executions

# cuda
if torch.cuda.is_available():
    dev = 'cuda:0'
else:
    dev = "cpu"
device = torch.device(dev)

def step(i):
    c = torch.ones(1)
    # comment the line below to see no memory increase
    c = c.to(device)
    time.sleep(sleep_time)

if __name__ == '__main__':
    arr = torch.ones((dim, dim))

    # create list of inputs to be executed in parallel
    inp = list(range(npe))

    # sleep added before and after launching multiprocessing to monitor the memory consumption
    print('before pool')  # to check memory with top or htop
    time.sleep(sleep_time)

    context = get_context('spawn')
    with context.Pool(npe) as pool:
        print('after pool')  # to check memory with top or htop
        time.sleep(sleep_time)

        pool.map(step, inp)

    time.sleep(sleep_time)


title: Target 40 is out of bounds for nn.CrossEntropyLoss()

body:
I create a custom image data set like:

from torch.utils.data.dataset import Dataset
from PIL import Image
import torchvision
from torchvision import datasets, models, transforms
import numpy as np

class MyCustomDataset(Dataset):
    def __init__(self, df, transforms=None):

        """
        Args:
            df (pandas.DataFrame): path to images and in dataframe
            transforms: pytorch transforms for transforms and tensor conversion
        """
        # Transforms
        self.transforms = transforms
        # Read the dataframe
        self.data_info = df
        # First column contains the image paths
        self.image_arr = np.asarray(self.data_info.iloc[:, 0])
        # Second column is the labels
        self.label_arr = np.asarray(self.data_info.iloc[:, 1])
        # Calculate len
        self.data_len = len(self.data_info.index)

    def __getitem__(self, index):
        # Get image name from the pandas df
        single_image_name = self.image_arr[index]
        # Open image
        img_as_img = Image.open(single_image_name)

        img_as_tensor= self.transforms(img_as_img)
        # Get label(class) of the image based on the cropped pandas column
        single_image_label = self.label_arr[index]

        return (img_as_tensor, single_image_label)

    def __len__(self):
        return self.data_len

The input of df for MyCustomDataset(df, transforms) is pandas dataframe storing images' paths and labels look like below:

    file_name                          label
0   M:\RealModels\images\001\001001.png 0
1   M:\RealModels\images\001\002001.png 0
2   M:\RealModels\images\001\003001.png 0
3   M:\RealModels\images\001\004001.png 0
4   M:\RealModels\images\001\006001.png 0
... ... ...
3197    M:\RenderedModels\images_rgb\450\116450.png 45
3198    M:\RenderedModels\images_rgb\450\117450.png 45
3199    M:\RenderedModels\images_rgb\450\118450.png 45
3200    M:\RenderedModels\images_rgb\450\119450.png 45
3201    M:\RenderedModels\images_rgb\450\120450.png 45
3202 rows × 2 columns

There are 16 classes in my dataset. The classes lables are like:['00', '01', '12', '34','35'...,'45']

My whole program is:

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
import time
import os
import copy
import scipy
from torch.utils.tensorboard import SummaryWriter

from torch.utils.data.dataset import Dataset
import torchvision
from torchvision import datasets, models, transforms

# In[2]:

import sys
sys.path.append(r"M:\program\pytorch\Scripts")
import custom_fun
from custom_fun import custom_dataset
import create_folder
from create_folder import create_tb_folder
import dataset_from_image
from dataset_from_image import MyCustomDataset

# In[3]:

tb_dir = r'../'
path = create_tb_folder(tb_dir)

# In[4]:

mean = np.array([0.485, 0.456, 0.406])
std = np.array([0.229, 0.224, 0.225])

# In[5]:

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomRotation(degrees=15),
        transforms.Resize((224,224)), # 299 for Inception v3
        transforms.ColorJitter(),
        transforms.ToTensor(),
        transforms.Normalize(mean, std)
    ]),
    'val': transforms.Compose([
        transforms.Resize((224,224)),
        transforms.ToTensor(),
        transforms.Normalize(mean, std)
    ]),
}

# In[6]:

data_dir = r'M:\dataset\first_att'

# In[7]:

batch_size = 4
lr = 0.003

# In[8]:

data = custom_dataset(data_dir,0.8)

# In[37]:

df = data['train']
df.head()

# In[9]:

image_datasets = {x: MyCustomDataset(data[x], data_transforms[x]) for x in ['train', 'val']}

# In[24]:

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size,
                                             shuffle=True, num_workers=0)
              for x in ['train', 'val']}

# In[25]:

dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
device = torch.device("cpu")

# In[26]:

# Get a batch of training data
inputs, classes = next(iter(dataloaders['train']))
print(inputs.shape,classes.shape)
print(classes)

# Make a grid from batch
out = torchvision.utils.make_grid(inputs)

# In[27]:

tb1 = SummaryWriter(path[1])
tb2 = SummaryWriter(path[2])

# In[28]:

def train_model(model, criterion, optimizer, scheduler, num_epochs):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        optimizer.zero_grad()
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase] # images of training data
            epoch_acc = running_corrects.double() / dataset_sizes[phase] # images of val data

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # Tensorboard works here
            if phase == 'train':
                tb1.add_scalar('Loss', epoch_loss, epoch)
                tb1.add_scalar('Accuracy', epoch_acc, epoch)
                for name, weight in model.named_parameters():
                    tb.add_histogram(name, weight, epoch)
                    tb.add_histogram(f'{name}.grad', weight.grad, epoch)

            else:
                tb2.add_scalar('Loss', epoch_loss, epoch)
                tb2.add_scalar('Accuracy', epoch_acc, epoch)

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

# In[29]:

model = models.resnet18(pretrained=True)

# The way below is feature extraction.
#for param in model.parameters():
    #param.requires_grad = False

# In[30]:

num_ftrs = model.fc.in_features

# In[31]:

model.fc = nn.Linear(num_ftrs, 16)

# In[32]:

model = model.to(device)
criterion = nn.CrossEntropyLoss()

# In[33]:

# Observe that all parameters are being optimized
optimizer = optim.Adam(model.parameters(), lr=lr)

# In[34]:

step_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

# In[35]:

tb = SummaryWriter(path[0])
grid = torchvision.utils.make_grid(inputs)
tb.add_image('images', grid)
tb.add_graph(model, inputs)
tb.close()

# In[36]:

model = train_model(model, criterion, optimizer, step_lr_scheduler, num_epochs=20)

# In[ ]:

torch.save(model.state_dict(), 'first_att_02.pth')

# In[ ]:

But I got the error:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-36-edba91d1cb93> in <module>
----> 1 model = train_model(model, criterion, optimizer, step_lr_scheduler, num_epochs=20)

<ipython-input-28-662f652902cf> in train_model(model, criterion, optimizer, scheduler, num_epochs)
     29                     outputs = model(inputs)
     30                     _, preds = torch.max(outputs, 1)
---> 31                     loss = criterion(outputs, labels)
     32
     33                     # backward + optimize only if in training phase

M:\program\pytorch\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

M:\program\pytorch\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

M:\program\pytorch\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

M:\program\pytorch\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target 40 is out of bounds.

Could you let me know where I am wrong? Thanks very much.


title: How does one use torch.optim.lr_scheduler.OneCycleLR()?

body:
I wanted to use torch.optim.lr_scheduler.OneCycleLR() while training. Can some kindly explain to me how to use it?
What i got from the documentation was that it should be called after each train_batch.
My confusions are as follows:

Does the max_lr parameter has to be same with the optimizer lr parameter?

Can this scheduler be used with Adam optimizer. How is the momentum calculated then?

Let’s say i trained my model for some number of epochs at a stretch now, i wanted to train for some more epochs. Would i have to reset the the scheduler?

Can anybody provide me a sort of a toy example/training loop that implements this scheduler?
I am kind of new to deep learning & PyTorch so my question might be somewhat silly.


title: Why can I install pytorch with pip but not with pipenv

body:
I am a relative pipenv newbie, and I am trying to install pytorch with cuda 11 for a machine learning project.
I am installing pytorch on linux with python 3.6. I have made a fresh pipenv environment with pipenv install.
To install the latest version of pytorch with cuda 11, the command is:
pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

When I run this command, it works fine and installs:
$ pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.7.0+cu110
  Using cached https://download.pytorch.org/whl/cu110/torch-1.7.0%2Bcu110-cp36-cp36m-linux_x86_64.whl (1137.1 MB)
Collecting torchvision==0.8.1+cu110
  Using cached https://download.pytorch.org/whl/cu110/torchvision-0.8.1%2Bcu110-cp36-cp36m-linux_x86_64.whl (12.9 MB)
Collecting torchaudio===0.7.0
  Using cached torchaudio-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (7.6 MB)
Collecting dataclasses
  Using cached dataclasses-0.8-py3-none-any.whl (19 kB)
Collecting numpy
  Using cached numpy-1.19.4-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)
Processing /home/denisonc/.cache/pip/wheels/6e/9c/ed/4499c9865ac1002697793e0ae05ba6be33553d098f3347fb94/future-0.18.2-py3-none-any.whl
Collecting typing-extensions
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting pillow>=4.1.1
  Using cached Pillow-8.0.1-cp36-cp36m-manylinux1_x86_64.whl (2.2 MB)
Installing collected packages: dataclasses, numpy, future, typing-extensions, torch, pillow, torchvision, torchaudio
Successfully installed dataclasses-0.8 future-0.18.2 numpy-1.19.4 pillow-8.0.1 torch-1.7.0+cu110 torchaudio-0.7.0 torchvision-0.8.1+cu110 typing-extensions-3.7.4.3
(deep-sinter) [denisonc@cvgpu02 deep-sinter]$ pipenv update
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (ca72e7)!
Installing dependencies from Pipfile.lock (ca72e7)...
     ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
All dependencies are now up-to-date!

However, changing 'pip' to 'pipenv' causes the following error:
$ pipenv install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
Installing torch==1.7.0+cu110...
Error:  An error occurred while installing torch==1.7.0+cu110!
Error text:
ERROR: Could not find a version that satisfies the requirement torch==1.7.0+cu110 (from -r /tmp/pipenv-tvts06b7-requirements/pipenv-0wd75ill-requirement.txt (line 1)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0)
ERROR: No matching distribution found for torch==1.7.0+cu110 (from -r /tmp/pipenv-tvts06b7-requirements/pipenv-0wd75ill-requirement.txt (line 1))
✘ Installation Failed

Last week I was able to complete this installation on the same system in a different pipenv. In the mean time I upgraded pipenv to the latest version. This error occurred after the upgade.
This is very confusing to me because my understanding was that pipenv uses pip under the hood. I can't understand why the behavior would be different. I would like to install pytorch within my pip environment so that I can keep libraries isolated from other projects.


title: PyToch Skorch passing 3 dimensional input

body:
Im trying to fit a model in PyTorch while using skorch. My problem is, that my model uses an LSTM layer, which expects 3d input and I don't know how to pass the input correctly.

When passing a 2d-array to fit() I get an error from PyTorch for the expected 3d input. If passing a 3d-array, I get the error from the fit() method for inconsistent lengths (which both make perfectly sense to me).

Example code below:

import numpy as np
import torch
from torch import nn
import skorch

class lstmNet(nn.Module):

    def __init__(self, input_size, hidden_size, num_layers):
        super(lstmNet, self).__init__()
        self.rnn = nn.LSTM(
            input_size,
            hidden_size,
            num_layers        )
        self.lin = nn.Linear(
            in_features=hidden_size,
            out_features=1
        )

    def forward(self, x):
        print(x.size())
        out, hn = self.rnn(x)
        out = self.lin(out)
        return out

input_feat = 5
hidden_size = 10
lstmLayers = 2

seq = 20
batch = 30
features = 5

net = skorch.NeuralNet(
    module=lstmNet(
        input_size=input_feat,
        hidden_size=hidden_size,
        num_layers=lstmLayers
    ),
    criterion=torch.nn.MSELoss,
    optimizer=torch.optim.SGD,
    lr=0.1,
    max_epochs=10
)

#inputArr2d = np.random.rand(seq * batch, features)
inputArr3d = np.random.rand(seq, batch, features)
print('input:\n {}\nshape: {}'.format(inputArr3d, inputArr3d.shape))
targetArr = np.random.rand((seq * batch))
#print('target:\n {}\nshape: {}'.format(targetArr, targetArr.shape))

net.fit(X=inputArr3d, y=targetArr)

This is the error when calling net.fit(X=inputArr2d, y=targetArr):

Traceback (most recent call last):
File "C:\Spielplatz\Python\examples\playground.py", line 64, in <module>
net.fit(X=inputArr2d, y=targetArr)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 686, in fit
self.partial_fit(X, y, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 646, in partial_fit
self.fit_loop(X, y, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 584, in fit_loop
step = self.train_step(Xi, yi, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 507, in train_step
y_pred = self.infer(Xi, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 810, in infer
return self.module_(x, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "C:\Spielplatz\Python\examples\playground.py", line 33, in forward
out, hn = self.rnn(x)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 178, in forward
self.check_forward_args(input, hx, batch_sizes)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 126, in check_forward_args
expected_input_dim, input.dim()))
RuntimeError: input must have 3 dimensions, got 2

And this is the error when calling net.fit(X=inputArr3d, y=targetArr):

Traceback (most recent call last):
File "C:\Spielplatz\Python\examples\playground.py", line 64, in <module>
net.fit(X=inputArr3d, y=targetArr)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 686, in fit
self.partial_fit(X, y, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 646, in partial_fit
self.fit_loop(X, y, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 573, in fit_loop
X, y, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 1004, in get_split_datasets
dataset = self.get_dataset(X, y)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\net.py", line 961, in get_dataset
return dataset(X, y, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\skorch\dataset.py", line 104, in __init__
raise ValueError("X and y have inconsistent lengths.")
ValueError: X and y have inconsistent lengths.


title: How do I add Poisson noise to a greyscale image in Python using numpy?

body:
I'm trying to implement adding Poisson noise to a greyscale image using numpy as a Pytorch transformer but so far my results have been very disappointing. Here are my various implementations with results ranging from seeing no visible changes to only seeing noise without an image. What is the correct way of doing this?

class Noise(object):
  def __init__(self, noise_type):
    self.noise_type = noise_type

  def __call__(self, sample):
    noise = np.random.poisson(1, sample.shape)
    return sample + noise

but as I understand it, Poisson noise can't be simply added to an image which is why all I see here is noise (kind of like TV static) without the original greyscale image.

The following version of the code yields no visible changes in the image which leads me to believe it's some sort of scaling issue since in Poisson noise, noise is added relative to a pixel's value.

vals = len(np.unique(sample))
vals = 2 ** np.ceil(np.log2(vals))
noisy = np.random.poisson(sample * vals) / float(vals)
return noisy

Any help or insight would be greatly appreciated.