title: Strange semantic segmentation results with UNet in Pytorch

body:
I was just trying to train UNet from scratch with a mammography dataset to detect tumor tissue in mammograms. after 35 training the model I got weird curves and bad results on the train and test set. I couldn't find any obvious mistake in my code, and everything seems correct.
Here is the link to implementation on GitHub: GitHub - hrnademi/Mammography
Further information about the dataset:
Training images: 2400 images with the size of 256x256 png
Test images: 600 images with the size of 256x256 png
Here are the curves:

Many thanks for your attention.


title: Questions about programming a cnn with PyTorch

body:
I'm pretty new at programming cnn so I'm a little bit lost. I'm trying to do this part of the code, where they ask me to implement a fully-connected network to classify the digits. It should contain 1 hidden layer with 20 units. I should use ReLU activation function on the hidden layer.
class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.fc1 = ...

        self.fc2 = nn.Sequential(
            nn.Linear(500,10),
            nn.Softmax(dim = 1)
            )

    def forward(self, x):
        x = x.view(x.size(0),-1)
        x = self.fc1(x)
        x = self.fc2(x)
        return x

The dots are the part to fill, I think about this line:
self.fc1 = nn.Linear(20, 500)

But I don't know if it's correct. Could someone help me please? And I don't understand at all what the function Softmax do... so if someone knows it please.
Thank you so much!!
Pd. This is the code to load the data:
batch_size = 64
trainset = datasets.MNIST('./data', train=True, download=True, transform=transforms.ToTensor())
train_loader = DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=1)
testset = datasets.MNIST('./data', train=False, download=True, transform=transforms.ToTensor())
test_loader = DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=1)


title: Image classification- Why am I getting vastly different results training on Tensorflow vs Pytorch?

body:
I am training an image classifier for a robot that detects "blocked" vs "free" paths. Using the same data on pytorch gives >0.98 accuracy on validation data whereas tensorflow only gives around 0.50-0.60 accuracy with a mode of 52.17%.
Pytorch is using pre-trained AlexNet implementation for which there is no counterpart on tensorflow. I tried my best to mirror the implementation on tensorflow as you can see below. I have also tried other CNN models including ResNet and InceptionV3 but all give me roughly the same validation accuracy on tensorflow.
Other information:

Input images are (224, 224, 3)
Both implementations point to same training and validation directories
I've tested both models with 4 images they have not seen before and pytorch model guessed correctly whereas tensorflow model did not.

Pytorch Code:
train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=8,
    shuffle=True,
    num_workers=0
)

test_loader = torch.utils.data.DataLoader(
    test_dataset,
    batch_size=8,
    shuffle=True,
    num_workers=0
)

model = models.alexnet(pretrained=True)
model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, 2)

device = torch.device('cuda')
model = model.to(device)

NUM_EPOCHS = 30
BEST_MODEL_PATH = 'best_model.pth'
best_accuracy = 0.0

optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

for epoch in range(NUM_EPOCHS):

    for images, labels in iter(train_loader):
        images = images.to(device)
        labels = labels.to(device)
        optimizer.zero_grad()
        outputs = model(images)
        loss = F.cross_entropy(outputs, labels)
        loss.backward()
        optimizer.step()

    test_error_count = 0.0
    for images, labels in iter(test_loader):
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        test_error_count += float(torch.sum(torch.abs(labels - outputs.argmax(1))))

    test_accuracy = 1.0 - float(test_error_count) / float(len(test_dataset))
    print('%d: %f' % (epoch, test_accuracy))
    if test_accuracy > best_accuracy:
        torch.save(model.state_dict(), BEST_MODEL_PATH)
        best_accuracy = test_accuracy

Tensorflow code:
model = keras.models.Sequential([
    keras.layers.Conv2D(filters=64, kernel_size=(11, 11), strides=(4, 4), activation='relu',
                        input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3)),
    keras.layers.BatchNormalization(),
    keras.layers.MaxPool2D(pool_size=(3, 3), strides=(2, 2)),
    keras.layers.Conv2D(filters=192, kernel_size=(5, 5), strides=(1, 1), activation='relu', padding="same"),
    keras.layers.MaxPool2D(pool_size=(3, 3), strides=(2, 2)),
    keras.layers.Conv2D(filters=384, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding="same"),
    keras.layers.Conv2D(filters=256, kernel_size=(1, 1), strides=(1, 1), activation='relu', padding="same"),
    keras.layers.Conv2D(filters=256, kernel_size=(1, 1), strides=(1, 1), activation='relu', padding="same"),
    keras.layers.MaxPool2D(pool_size=(3, 3), strides=(2, 2)),
    keras.layers.Dropout(0.5),
    keras.layers.Flatten(),
    keras.layers.Dense(4096, activation='relu'),
    keras.layers.Dropout(0.5),
    keras.layers.Dense(4096, activation='relu'),
    keras.layers.Dense(1, activation='sigmoid')
])

train_gen = ImageDataGenerator(
    rescale=1.0 / 255,
    samplewise_std_normalization=True,
    # rotation_range=20,
    # width_shift_range=0.2,
    # height_shift_range=0.1,
    brightness_range=(0.0, 0.5),
    # shear_range=0.2,
    zoom_range=0.3,
    fill_mode='nearest'
)

val_gen = ImageDataGenerator(rescale=1.0 / 255)

train_data = train_gen.flow_from_directory(
    train_path,
    target_size=(IMAGE_SIZE, IMAGE_SIZE),
    class_mode='binary',
    color_mode='rgb',
    batch_size=8,
    shuffle=True)

val_data = val_gen.flow_from_directory(
    val_path,
    target_size=(IMAGE_SIZE, IMAGE_SIZE),
    class_mode='binary',
    color_mode='rgb',
    batch_size=8,
    shuffle=True)

optimizer = tf.keras.optimizers.SGD(lr=0.001, momentum=0.9)
loss = tf.keras.losses.binary_crossentropy
model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])
num_epochs = 40
save_path = 'F:\\Documents\\Jetbot\\collision_avoidance_tf\\model_save'
model_checkpoint = tf.keras.callbacks.ModelCheckpoint(save_path, monitor='val_loss', save_best_only=True)
history = model.fit(train_data, epochs=num_epochs, validation_data=val_data, verbose=1,
                    callbacks=[model_checkpoint])

Any help is greatly appreciated. Thanks!


title: torch: minimally pad tensor such that num elements divisible by x

body:
Suppose I have a tensor t of arbitrary ndim
I want to pad (with zeroes) it such that
a) I introduce the fewest possible # elements
b) after padding, (t.numel() % x) == 0
Is there a better algorithm for doing this than
find the largest dimension and increase it by 1 until condition (b) is satisfied?
Maybe working code:
def pad_minimally(t, x):
    largest_dim = np.argmax(t.shape)
    buffer_shape = list(t.shape)
    new_t = t.clone()
    print(t.shape)
    for n_to_add in range(x):
        if new_t.numel() % x == 0:
            break
        buffer_shape[largest_dim] = n_to_add
        new_buffer = torch.zeros(*buffer_shape)
        new_t = torch.cat([t, new_buffer], axis=largest_dim)
    assert new_t.numel() % x == 0
    return new_t
assert pad_minimally(torch.rand(3,1), 7).shape == (7,1)
assert pad_minimally(torch.rand(3,2), 7).shape == (7,2)
assert pad_minimally(torch.rand(3,2, 6), 7).shape == (3,2,7)


title: BERT Convert 'SpanAnnotation' to answers using scores from hugging face models

body:
I'm following along with the documentation for importing a pretrained model question and answer model from huggingface
from transformers import BertTokenizer, BertForQuestionAnswering
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-base-uncased')
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
inputs = tokenizer(question, text, return_tensors='pt')
start_positions = torch.tensor([1])
end_positions = torch.tensor([3])
outputs = model(**inputs, start_positions=start_positions, end_positions=end_positions)
loss = outputs.loss
start_scores = outputs.start_logits
end_scores = outputs.end_logits

this returns start and end scores, but how can I get a meaningful text answer from here?


title: Embedding in PyTorch creates embedding with norm larger than max_norm

body:
Suppose we have an embedding matrix of 10 vectors with dimension of 100, and we impose max_norm=1:
x = Embedding(num_embeddings=10, embedding_dim=100, max_norm=1)

In principle, every embedding should have norm less or equal to 1. However, when I print the vector norms, I get values much greater than 1:
for w in x.weight:
    print(torch.norm(w))

> tensor(11.1873, grad_fn=<CopyBackwards>)
> tensor(10.5264, grad_fn=<CopyBackwards>)
> tensor(9.6809, grad_fn=<CopyBackwards>)
> tensor(9.7507, grad_fn=<CopyBackwards>)
> tensor(10.7940, grad_fn=<CopyBackwards>)
> tensor(11.4134, grad_fn=<CopyBackwards>)
> tensor(9.7021, grad_fn=<CopyBackwards>)
> tensor(10.4027, grad_fn=<CopyBackwards>)
> tensor(10.1210, grad_fn=<CopyBackwards>)
> tensor(10.4552, grad_fn=<CopyBackwards>)

Any particular reason why this happens and how to fix it?


I am trying to perform this operation:
L_c += 1 - torch.abs(torch.dot(nuNormalized, Nu))

where:

nuNormalized -> tensor([ 0.3790, -0.4208,  0.8242], dtype=torch.float64)

Nu -> tensor([-0.9961, -0.9961, -0.9961], device='cuda:0')

I am getting this error:

(<class 'RuntimeError'>, RuntimeError('dot : expected both vectors to have same dtype, but found Double and Float',), <traceback object at 0x7f4d276155c8>)

Any suggestions please? Thanks!


title: "Visualizing [and Understanding] Recurrent Networks" using LSTM's cell state at each timestep

body:
I am currently trying to programmatically visualize the way deep learning approaches learn/train on my problems and e.g. stumbled upon the paper "Visualizing and Understanding Recurrent Networks" by Andrej Karpathy, Justin Johnson, and Li Fei-Fei (e.g. https://arxiv.org/abs/1506.02078).
In figure 2 they calculate the tanh of the memory/cell states (c) of their LSTMs at each timestep during prediction.
I now want to do the same with my LSTMs in Keras (or PyTorch, but I'm quite new to their API). The API allows me to get the hidden states (h) for each timestep - using the parameter return_sequences=True, but only the last cell state - using the parameter return_state=True.
With my understanding of the Keras API I currently create two models:
inputs = Input(shape=(MAX_LEN,))

embd = Embedding(input_dim=MAX_FEATURES, output_dim=128, input_length=MAX_LEN)(inputs)
lstm = LSTM(units=128, return_sequences=True)(embd)

# 'ignore' return_sequences=True in training model
flt = Flatten()(lstm)

drp = Dropout(0.5)(flt)
dns = Dense(N_CLASSES)(drp)
activ = Activation('softmax')(dns)

training_model = Model([inputs], [activ])
state_model = Model([inputs], [activ, lstm])

training_model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
training_model.fit(X_train, y_train)

predictions, hidden_states = state_model.predict(X_test)

How do I do the same for the cell state at each timestep (character in my case) - preferably without having to override/reimplement 'major' parts of Keras'/PyTorch's source code?
I found this answer, but haven't gotten it to work and as I don't have enough reputation I can't add any questions in the comments (I saw the other answer but I want to prevent a 'complete' reimplementation of Keras' LSTMCell).
EDIT #1: As I'm quite new to PyTorch's API - how would I go about there?


title: How to convert a CNN LSTM form keras to pytorch

body:
I am trying to convert a CNN LSTM for keras to pytorch but I have trouble.
ConvNN_model = models.Sequential()
ConvNN_model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(64, 64, 1)))
ConvNN_model.add(layers.MaxPooling2D((2, 2)))
ConvNN_model.add(layers.Conv2D(64, (3, 3), activation='relu'))
ConvNN_model.add(TimeDistributed(LSTM(128, activation='relu')))
ConvNN_model.add(Dropout(0.2))
ConvNN_model.add(LSTM(128, activation='relu'))
ConvNN_model.add(layers.Dense(64, activation='relu'))
ConvNN_model.add(layers.Dropout(0.25))
ConvNN_model.add(layers.Dense(15, activation='softmax'))

How to convert the above code from Keras to Pytorch?


title: torch concat 1D to a 2D tensor

body:
I have a issue on concat 2 tensor,
say I have x and y:
x = torch.randn(35, 50)
y = torch.randn(35)

How do I concat every y value in to x[0] to make x has a shape 35,51?
I tried:
for i in y:
    for a in range(x.shape[0]):
        x[a] = torch.cat((x[a],i),0)

Still getting shape error. Any smart way of doing it?


title: Pytorch: smarter way to reduce dimension by reshape

body:
I want to reshape a Tensor by multiplying the shape of first two dimensions.
For example,
1st_tensor: torch.Size([12, 10]) to torch.Size([120])
2nd_tensor: torch.Size([12, 10, 5, 4]) to torch.Size([120, 5, 4])
I.e. The first two dimensions shall be merged into one, while the other dimensions shall remain the same.
Is there a smarter way than
1st_tensor.reshape(-1,)
2nd_tensor.reshape(-1,5,4),
that can adapt to the shape of different Tensors?

Test cases:
import torch
tests = [
    torch.rand(11, 11),
    torch.rand(12, 15351, 6, 4),
    torch.rand(13, 65000, 8)
    ]


title: PyTorch: Why create multiple instances of the same type of layer?

body:
This code is from PyTorch transformer:
    self.linear1 = Linear(d_model, dim_feedforward, **factory_kwargs)
    self.dropout = Dropout(dropout)
    self.linear2 = Linear(dim_feedforward, d_model, **factory_kwargs)
    self.norm1 = LayerNorm(d_model, eps=layer_norm_eps, **factory_kwargs)
    self.norm2 = LayerNorm(d_model, eps=layer_norm_eps, **factory_kwargs)
    self.norm3 = LayerNorm(d_model, eps=layer_norm_eps, **factory_kwargs)
    self.dropout1 = Dropout(dropout)
    self.dropout2 = Dropout(dropout)
    self.dropout3 = Dropout(dropout)

Why do they add self.dropout1, ...2, ...3 when self.dropout already exists and is the exact same function?
Also, what is the difference between (self.linear1, self.linear2) and self.linear?


title: Reverse operation of torch.cat

body:
Suppose I have a tensor like [A,A,A,A...,A].
How can I quickly obtain [[A],[A],[A],...,[A]] as a tensor in torch?


title: FileNotFoundError: [Errno 2] :No such file or directory: 'C:/Users/My_computer/Desktop/Compare/MHAN-master/AID_train/AID_train_LR/x4\\9.png'

body:
I'm very new to python environment. I have tried to compile a super-resolution code for upscaling factor 4 using my own dataset. The low resolution RGB images are kept in "C:/Users/My_computer/Desktop/Compare/MHAN-master/AID_train/AID_train_LR/x4". The code used for image load is shown in below:
def load_img(filepath):
img = Image.open(filepath).convert('RGB')
#img = Image.open(filepath, 'rb')

#y, _, _ = img.split()
return img

class DatasetFromFolder(data.Dataset):
def __init__(self, image_dir, lr_dir, patch_size, upscale_factor, data_augmentation, transform=None):
    super(DatasetFromFolder, self).__init__()
    self.image_filenames = [join(image_dir, x) for x in listdir(image_dir) if is_image_file(x)]
    self.patch_size = patch_size
    self.upscale_factor = upscale_factor
    self.transform = transform
    self.data_augmentation = data_augmentation
    self.HR ='C://Users//My_computer//Desktop//Compare//MHAN-master//AID_train//AID_train_HR'
    self.LR ='C://Users//My_computer//Desktop//Compare//MHAN-master//AID_train//AID_train_LR//x4'

def __getitem__(self, index):
    target = load_img(self.image_filenames[index])
    input = load_img(os.path.join(self.LR, file))
    input, target, _ = get_patch(input,target,self.patch_size, self.upscale_factor)
    return input, target

But I am getting the following error while the training code is compiled:
File "main_x4.py", line 185, in <module>
    train(model, epoch)
  File "main_x4.py", line 60, in train
    for iteration, batch in enumerate(training_data_loader, 1):
  File "C:\Users\My_computer\anaconda3\envs\MHAN\lib\site-packages\torch\utils\data\dataloader.py", line 346, in __next__
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\My_computer\anaconda3\envs\MHAN\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\My_computer\anaconda3\envs\MHAN\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\My_computer\Desktop\Compare\MHAN-master\dataset_x4.py", line 91, in __getitem__
    input = load_img(os.path.join(self.LR, file))

File "C:\Users\My_computer\Desktop\Compare\MHAN-master\dataset_x4.py",
line 16, in load_img

img = Image.open(filepath).convert('RGB')

  File "C:\Users\My_computer\anaconda3\envs\MHAN\lib\site-packages\PIL\Image.py",

line 2912, in open
fp = builtins.open(filename, "rb")

FileNotFoundError: [Errno 2] No such file or directory: 'C://Users//My_computer//Desktop//Compare//MHAN-master//AID_train//AID_train_LR//x4\\9.png'

As LR images are already in RGB format, so is it necessary to convert to RGB again?
Please help me to fix this error


title: fblualib installation in Ubunt16.04

body:
I am trying to install fblualib from this link.

When I run ./install_all.sh inside fblualib, I have errors as

+ echo Building folly
Building folly
+ echo

+ cd /tmp/fblualib-build.zofU4g/folly/folly
+ autoreconf -ivf
autoreconf: 'configure.ac' or 'configure.in' is required

According to last discussion here, there is a discussion as "autotools based build system has been removed from Folly in favor of CMake, so you'll need to build Folly with CMake rather than autotools." so I go to folly folder

/tmp/fblualib-build.zofU4g/folly/build and try to build using `cmake`.

Then I have errors as

[ 36%] Building CXX object CMakeFiles/folly_base.dir/folly/detail/ThreadLocalDetail.cpp.o
In file included from /tmp/fblualib-build.zofU4g/folly/folly/detail/ThreadLocalDetail.h:33:0,
                 from /tmp/fblualib-build.zofU4g/folly/folly/detail/ThreadLocalDetail.cpp:16:
/tmp/fblualib-build.zofU4g/folly/folly/SharedMutex.h: In static member function ‘static void folly::threadlocal_detail::StaticMetaBase::onThreadExit(void*)’:
/tmp/fblualib-build.zofU4g/folly/folly/SharedMutex.h:457:33: error: ‘*((void*)& rlock +10)’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
         !tryUnlockSharedDeferred(token.slot_)) {
                                 ^
/tmp/fblualib-build.zofU4g/folly/folly/detail/ThreadLocalDetail.cpp:169:31: note: ‘*((void*)& rlock +10)’ was declared here
       SharedMutex::ReadHolder rlock(nullptr);
                               ^
cc1plus: all warnings being treated as errors
CMakeFiles/folly_base.dir/build.make:1886: recipe for target 'CMakeFiles/folly_base.dir/folly/detail/ThreadLocalDetail.cpp.o' failed
make[2]: *** [CMakeFiles/folly_base.dir/folly/detail/ThreadLocalDetail.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/folly_base.dir/all' failed
make[1]: *** [CMakeFiles/folly_base.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

The error is happening in SharedMutex.h, what could be worng>


title: How do I assign a matrix of data as a label to each input image in my dataset using PyTorch?

body: I want to train a convolutional neural network (CNN) in PyTorch to predict frequency spectrum data related to an input image. Rather than assigning one label to each image (Dog, Cat, Car, Airplane, etc.), I would like to assign a matrix of labels (one label per frequency) to each image. In PyTorch, how do I assign a matrix of data as a label to each input image in my dataset? I have been trying to do this using ImageFolder. Thanks!


title: In binary classification, when evaluating in .train() mode my model gets a high accuracy on my test data

body: I use pytorch train a binary model. When evaluating in model.train() mode my model gets a high accuracy on my test data than model.eval().I used the BN layer


title: How to calculate meters in Pytorch1.1 & DistributedDataParallel()?

body:
I want to use model parallel and data parallel at the same time, and have read many docs and tutorials from official website.
One confusing problem I faced is how to collect all kinds of meter values in each Process?

Question1: In the official tutorial, they just record meters value in each Process.
But in my code, I print loss value in each process, they are different. So, I think the value of other meters are also different.
Is that tutorial wrong? In my opinion, I think the right way should synchronize loss, acc and other meters first, then all processes maintain the same values, after that I just need to print meters information in one Process.

Question2: In the official tutorial, they say 'the DistributedDataParallel module also handles the averaging of gradients across the world, so we do not have to explicitly average the gradients in the training step'.
But, because of question1, does the API actually work as what the tutorial said?  Because each of the processes has a different loss value, although they start from the same init weights, will model weights in each process be optimized in different directions?


title: Matplotlib Pylot - Images are being displayed in low resolution (pixel to pixel)

body:
When I display some samples photos from the dataset I use, the previews of images are displayed in low resolution (they look like very low-resolution photos). How I can I display the images without losing their resolutions?

Here are my transformations which are used to move the data to the tensor and apply some transformations using PyTorch functions:

data_transforms = transforms.Compose([
    transforms.Resize((50, 50)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])

Then I load the data through DataLoader:

train_loader = DataLoader(face_train_dataset,
                          batch_size=train_batch_size, shuffle=False,
                          num_workers=4)

Finally, I display some previews for the sample photos which are retrieved using the DataLoader object:

example_data = example_data.cpu()
example_targets = example_targets.cpu()
for i in range(6):
    plt.subplot(2, 3, i + 1)
    plt.tight_layout()
    plt.imshow(example_data[i][0], cmap='gray', interpolation='none')
    plt.title('{}'.format(folders[example_targets[i]]))

plt.show()

p.s. Images are in tiff format.


title: rebuild torch tensor from its batchified version

body:
This is very nice example of how to build a 3D tensor:

import torch
y = torch.rand(100, 1)
batch_size = 10
batched_data = y.contiguous().view(batch_size, -1, y.size(-1)).transpose(0,1)
batched_data.shape

the output is:

torch.Size([10, 10, 1])

ok, now what I want to do is, starting from batched_data I want to build y.
The other way around.
Any good suggestion with a powerful pytorch streamlined code?

==== Additional input =====

I am using this for RNN and now I have some doubts, becaus eif you consider the following code:

import torch
y = torch.arange(100).view(100,1)
batch_size = 10
batched_data = y.contiguous().view(batch_size, -1, y.size(-1)).transpose(0,1)
batched_data.shape

The output is:

tensor([[[ 0],
         [10],
         [20],
         [30],
         [40],
         [50],
         [60],
         [70],
         [80],
         [90]],

        [[ 1],
         [11],
         [21],
         [31],
         [41],
         [51],
         [61],
         [71],
         [81],
         [91]],

Which I would not expect. I would expect something like:
[[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20],....


title: Variable bug(s) O'Reilly Programming PyTorch

body:
I'm reading O'Reilly's Sept, 2019 publication 'Programming Pytorch ..' describing a simple linear neural network for image classification.

There is a bug in a variable name in the opening model (no worries) target vs. targets, however there is what appears to be a weird omission of a variable declaration, train_iterator (and also dev_iterator, not shown).

I wish to know is what was the train_iterator variable they (I presume) intended?

p27

def train(model, optimiser, loss_fn, train_loader, val_loader, epochs=20, device='cpu'):
    for epoch in range(epochs):
        training_loss = 0.0
        valid_loss = 0.0
        model.train()
        for batch in train_loader:
            optimizer.zero_grad()
            inputs, target = batch # Bug here for 'target'
            inputs = inputs.to(device)
            target = targets.to(device)
            output = model(inputs)
            loss = loss_fin(output, target)
            loss.backward()
            optimizer.step()
            training_loss += loss.data.item()
        training_loss /= len(train_iterator) # What is train_iterator?

so,..

    inputs, target = batch

must be

    inputs, targets = batch

In the validation step below the training step (not shown) is

    inputs, targets = batch
    ...
    targets = targets.to(device)

Its no biggie and the code is simply assigning to Cudas (GPU) or CPU.

The variable train_iterator is defining the training loss (important diagnostic). I assume that there should be an iterator declared between the epoch and the batch iterator, or is this within the training loop?

Notes train_loader simply refers Pytorch Dataloader. The model refers to 3 linear layers with a ReLU activation function.


title: How can I vectorize these nested loops in Python?

body:
I am trying to vectorize the following:

n = torch.zeros_like(x)
for i in range(x.shape[0]):
    for j in range(x.shape[1]):
        for k in range(x.shape[2]):
            n[i, j, k] = p[i, x[i, j, k], j, k]

I tried doing something like

n = p[:, x, ...]

but I just get an error that I ran out of memory, which isn't very helpful. I think the problem with this is that instead of getting the value of x at the correct index it is trying to index the entirety of x, but I am not sure how I would go about fixing that if that is the problem.


title: How to create properly shaped input using Dataloader?

body:
I am trying to input an image and a vector as input to the model. The image has the correct shape of 4d, but the vector that I input doesn't have such shape. The image size is 424x512 while the vector is of shape (18,). After using dataloader, I get batches of shape (50x1x424x512) and (50x18). Model gives error as it needs the vector shape to be 4d too. How do I do that?
Here is my code :

def loadTrainingData_B(args):
    fdm = []
    tdm = []
    parameters = []
    for i in image_files[:4]:
        try:
            false_dm = np.fromfile(join(ref, i), dtype=np.int32)
            false_dm = Image.fromarray(false_dm.reshape((424, 512, 9)).astype(np.uint8)[:,:,1])
            fdm.append(false_dm)
            true_dm = np.fromfile(join(ref, i), dtype=np.int32)
            true_dm = Image.fromarray(true_dm.reshape((424, 512, 9)).astype(np.uint8)[:,:,1])
            tdm.append(true_dm)
            pos = param_filenames.index(i)
            param = np.array(params[pos, 1:])
            param = np.where(param == '-point-light-source', 1, param).astype(np.float64)
            parameters.append(param)
        except:
            print('[!] File {} not found'.format(i))
    return (fdm, parameters, tdm)

class Flat_ModelB(Dataset):
    def __init__(self, args, train=True, transform=None):
        self.args = args
        if train == True:
            self.fdm, self.parameters, self.tdm = loadTrainingData_B(self.args)
        else:
            self.fdm, self.parameters, self.tdm = loadTestData_B(self.args)
        self.data_size = len(self.parameters)
        self.transform = transforms.Compose([transforms.ToTensor()])

    def __getitem__(self, index):
        return (self.transform(self.fdm[index]).double(), torch.from_numpy(self.parameters[index]).double(), self.transform(self.tdm[index]).double())

    def __len__(self):
        return self.data_size

The error I get is :

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 5 5, but got 2-dimensional input of size [50, 18] instead

Here is the model :

class Model_B(nn.Module):
    def __init__(self, config):
        super(Model_B, self).__init__()
        self.config = config
        # CNN layers for fdm
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(16))
        self.layer2 = nn.Sequential(
            nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer3 = nn.Sequential(
            nn.Conv2d(in_channels=32, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer4 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=32, out_channels=32, kernel_size=5, stride=2, padding=2, output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer5 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=32, out_channels=16, kernel_size=5, stride=2, padding=2,output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(16))
        self.layer6 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=16, out_channels=1, kernel_size=5, stride=2, padding=2, output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(1))
        # CNN layer for parameters
        self.param_layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))

    def forward(self, x, y):
        out = self.layer1(x)
        out_param = self.param_layer1(y)
        print("LayerParam 1 Output Shape : {}".format(out_param.shape))
        print("Layer 1 Output Shape : {}".format(out.shape))
        out = self.layer2(out)
        print("Layer 2 Output Shape : {}".format(out.shape))
        out = self.layer3(out)
        # out = torch.cat((out, out_param), dim=2)
        print("Layer 3 Output Shape : {}".format(out.shape))
        out = self.layer4(out)
        print("Layer 4 Output Shape : {}".format(out.shape))
        out = self.layer5(out)
        print("Layer 5 Output Shape : {}".format(out.shape))
        out = self.layer6(out)
        print("Layer 6 Output Shape : {}".format(out.shape))
        return out

and the method by which I access the data :

for batch_idx, (fdm, parameters) in enumerate(self.data):
            if self.config.gpu:
                fdm = fdm.to(device)
                parameters = parameters.to(device)
                print('shape of parameters for model a : {}'.format(parameters.shape))

            output = self.model(fdm)
            loss = self.criterion(output, parameters)

Edit :
I think my code is incorrect as I am trying to apply convolutions over a vector of (18). I tried to copy the vector and make it (18x64) and then input it. It still doesnt work and gives this output :

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 5 5, but got 3-dimensional input of size [4, 18, 64] instead

I am not sure how to concatenate an 18 length vector to the output of layer 3, if I can't do any of these things.


title: Why pytorch model cannot recognize the tensors I defined?

body:
I'm just learn pytorch recently.
And I try to write a same model like the paper that I have read for practice.

This is the PDF of the paper I refer.
https://dl.acm.org/doi/pdf/10.1145/3178876.3186066?download=true

Here is the code what I wrote.

class Tem(torch.nn.Module):
    def __init__(self, embedding_size, hidden_size):
        super(Tem, self).__init()
        self.embedding_size = embedding_size
        self.hidden_size = hidden_size
        self.leaf_size = 0

        self.xgb_model = None
        self.vec_embedding = None

        self.multi_hot_Q = None
        self.user_embedding = torch.nn.Linear(1, embedding_size)
        self.item_embedding = torch.nn.Linear(1, embedding_size)

    def pretrain(self, ui_attributes, labels):
        print("Start XGBoost Training...")
        self.xgb_model = XGBoost(ui_attributes, labels)
        self.leaf_size = self.xgb_model.leaf_size
        self.vec_embedding = Variable(torch.rand(self.embedding_size, self.leaf_size, requires_grad=True))

        self.h = Variable(torch.rand(self.hidden_size, 1, requires_grad=True))
        self.att_w = Variable(torch.rand(2 * self.embedding_size, self.hidden_size, requires_grad=True))
        self.att_b = Variable(torch.rand(self.leaf_size, self.hidden_size, requires_grad=True))

        self.r_1 = Variable(torch.rand(self.embedding_size, 1, requires_grad=True))
        self.r_2 = Variable(torch.rand(self.embedding_size, 1, requires_grad=True))
        self.bias = Variable(torch.rand(1, 1, requires_grad=True))

    def forward(self, ui_ids, ui_attributes):
        if self.xgb_model == None:
            raise Exception("Please run Tem.pretrain() to pre-train XGBoost model first.")

        n_data = len(ui_ids)

        att_input = torch.FloatTensor(ui_attributes)
        self.multi_hot_Q = torch.FloatTensor(self.xgb_model.multi_hot(att_input)).permute(0,2,1)

        vq = self.vec_embedding * self.multi_hot_Q

        id_input = torch.FloatTensor(ui_ids)

        user_embedded = self.user_embedding(id_input[:,0].reshape(n_data, 1))

        item_embedded = self.item_embedding(id_input[:,1].reshape(n_data, 1))

        ui = (user_embedded * item_embedded).reshape(n_data, self.embedding_size, 1)

        ui_repeat = ui.repeat(1, 1, self.leaf_size)

        cross = torch.cat([ui_repeat, vq], dim=1).permute(0,2,1)

        re_cross = corss.reshape(cross.shape[0] * cross.shape[1], cross.shape[2])

        attention = torch.mm(re_cross, self.att_w)
        attention = F.leaky_relu(attention + self.att_b.repeat(n_data, 1))
        attention = torch.mm(attention, self.h).reshape(n_data, self.leaf_size)
        attention = F.softmax(attention).reshape(n_data, self.leaf_size, 1)
        attention = self.vec_embedding.permute(1,0) * attention.repeat(1,1,20)

        pool = torch.max(attention, 1).values

        y_hat = self.bias.repeat(n_data, 1) + torch.mm(ui.reshape(n_data, self.embedding_size), self.r_1) + torch.mm(pool, self.r_2)
        y_hat = F.softmax(torch.nn.Linear(1, 2)(y_hat))
        return y_hat

My question is...It seems torch didn't know what tensor should be calculate gradient in backward propagation.

print(tem)

Tem(
(user_embedding): Linear(in_features=1, out_features=20, bias=True)
(item_embedding): Linear(in_features=1, out_features=20, bias=True)
)

I googled this problem, someone says those tensors should use torch.autograd.Variable(), but it didn't solve my problem. And someone says autograd directly supports tensors now. torch.autograd.Variable() is not necessary. Thanks a lot if you know how to solve it.

loss_func = torch.nn.CrossEntropyLoss()
optimizer = torch.Adagrad(tem.parameters(), lr=0.02)
for t in range(20):
    prediction = tem(ids_train, att_train)

    loss = loss_func(prediction, y_train)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if t % 5 == 0:
        print("loss: ", loss)

loss: tensor(0.8133, grad_fn=<NllLossBackward>)
loss: tensor(0.8133, grad_fn=<NllLossBackward>)
loss: tensor(0.8133, grad_fn=<NllLossBackward>)
loss: tensor(0.8133, grad_fn=<NllLossBackward>)


title: creating a common embedding for two languages

body:
My task deals with multi-language like (english and hindi). For that I need a common embedding to represent both languages.

I know there are methods for learning multilingual embedding like 'MUSE', but this represents those two embeddings in a common vector space, obviously they are similar, but not the same.

So I wanted to know if there is any method or approach that can learn to represent both embedding in form of a single embedding that represents the both the language.

Any lead is strongly appreciated!!!


title: How to append a list with arrays of two different sizes based on conditons

body:
I was wondering how to do this in a more efficient way for arbitrary arrays, the code is written in PyTorch, but it is only for 1-d tensors.
Thank you!

test=[]
data=np.random.uniform(0,1,[20,])
x=torch.from_numpy(data).float()
x,_=torch.sort(x)
v=torch.rand(5).float()
v,_=torch.sort(v)
for i in range(len(x)):
    if x[i] < v[0]:
         test.append(v[0])
    elif x[i] < v[1]:
         test.append(v[1])
    elif x[i] < v[2]:
           test.append(v[2])
    elif x[i] < v[3]:
           test.append(v[3])
    else:
        test.append(v[4])

  test


title: Image slider in tensorboard automatically goes to end while user scrolls through images

body:
I am using TensorBoard 2.2.0 to visualize my input image to a neural network in PyTorch. After running the code and I try to go through the recorded image data. However, when I try to browse it the image slider automatically jumps from the current step to the last step after some time. Any reason why this could be happening.

I am launching the tensorboard using this command -

tensorboard --logdir runs --samples_per_plugin images=0


title: Use a generator to perform operation on a matrix in Python

body:
I have a similarity matrix (torch tensor) which is a cosine similarity matrix between two matrix (source and target). From the matrix I need to obtain the sum of the distance between the top nearest neighbor of each source and target.

Then fillup two defaultdicts using the computed values above as shown in the code snippet below

import torch
from collections import defaultdict
src2tgt = defaultdict(dict)
tgt2src = defaultdict(dict)

#similarity matrix between source and target matrix
matx = torch.Tensor([[3,2,1,7],[1,1,0,8],[0,7,1,0],[2,0,0,0],[1,5,2,1]])

#the src and tgt
src = torch.LongTensor([[1,1],[1,2],[1,3],[1,4],[1,5]])
tgt = torch.LongTensor([[2,1],[2,2],[2,3],[2,4]])
#the data above are dummy, in my actual code, they are generated by a process
similarities=[]

#I need a kinda of the sum of nearest neighbor
k = 2
nearestSrc = torch.topk(matx, k, dim=1, largest=True, sorted=False, out=None)
sumDistSource = torch.sum(nearestSrc[0], 1)
nearestTgt = torch.topk(matx, k, dim=0, largest=True, sorted=False, out=None)
sumDistTarget = torch.sum(nearestTgt[0], 0)

#finally fill default dictionary of source2target and target2source
for i in range(matx.shape[0]):
  for j in range(matx.shape[1]):
    src2tgt[src[i]][tgt[j]] = matx[i][j].tolist() / (sumDistSource[i].tolist() + sumDistTarget[j].tolist())
    tgt2src[tgt[j]][src[i]] = matx[i][j].tolist() / (sumDistTarget[j].tolist() + sumDistSource[i].tolist())
    similarities.append(matx[i][j].tolist() )

Is there a way I can optimize the above code, either using a generator without having to create nearestSrc, sumDistSource, nearestTgt, sumDistTarget explicitly thus requiring less memory?  Or can I also reduce the double loop?


title: LSTM cannot predict well when the ground truth is near zero

body:
While training the LSTM model, I encountered one problem that I couldn't solve. To begin with, let me describe my model: I used a stacked LSTM model in Pytorch with 3 layers, 256 hidden units in each layer to predict human joint torques and joint angles from EMG features. After the training, the model can predict well when the ground truth is far away from 0, but when the ground truth is near zero, there is always an offset between the predicted value and the ground truth. I guess the reason would be that the large value of ground truth will give more impact during the training process to reduce the loss function.

This is the result:

The prediction for the validation set

The prediction for the training set

As you can see from the figures, in both datasets, the model can predict well when the ground truth is above 20 degrees. I have tried with different loss functions but the situation did not improve. Since I am just a beginner in this field, I hope someone can point out the problem in my method and how to solve it. Thank you!


title: Pytorch Error: Optimizer got an Empty Parameter List on Linux server

body:
I am running FasterRCNN code and was experimenting with various backbones (resnet18...101). I have created FasterRCNN class with init function. While training on my linux server I am facing the "Empty Parameter" error (can be seen in the attached image 1)

While the same code is running without any error on my local machine.
I am using Anaconda distribution 4.0.1 and Python 3.7 with pytorch 1.2
Below you can refer my code (model and train) -

class Faster_RCNN(nn.Module):
    def __init__(self, num_classes=2, backbone = 'mobilenet_v2', test= False):
        super(Faster_RCNN, self).__init__()
        self.classes = num_classes # replace the classifier with a new one, that has num_classes 2 for our use.
        # 1 class (bar) + background
        self.test = test
        self.backbone = backbone
        print('....Initializing Model....\n')
        print(f'....This model uses {self.backbone} as backbone....\n')
        if self.backbone == 'mobilenet_v2':
            backbone = torchvision.models.mobilenet_v2(pretrained=True).features
            backbone.out_channels = 1280
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)

        if self.backbone == 'resnet18':
            bb = torchvision.models.resnet18(pretrained=True)
            backbone = nn.Sequential(*list(bb.children())[:-2])
            backbone.out_channels = 512
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)

        if self.backbone == 'resnet34':
            bb = torchvision.models.resnet34(pretrained=True)
            backbone = nn.Sequential(*list(bb.children())[:-2])
            backbone.out_channels = 512
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)
        if self.backbone == 'resnet50':
            bb = torchvision.models.resnet50(pretrained=True)
            backbone = nn.Sequential(*list(bb.children())[:-2])
            backbone.out_channels = 2048
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)
        if self.backbone == 'resnet101':
            bb = torchvision.models.resnet50(pretrained=True)
            backbone = nn.Sequential(*list(bb.children())[:-2])
            backbone.out_channels = 2048
            anchor_generator = AnchorGenerator(sizes=((32, 64, 128),),
                                       aspect_ratios=((1.0),))
            roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
            self.model = FasterRCNN(backbone,
                       num_classes=2,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)

    def forward(self, x,y):
        if self.test:
            pred = self.model(x)
        else:
            pred = self.model(x,y)
        return pred

Here is the train code:

import dataloader
import model as md
import numpy as np
#import matplotlib.pyplot as plt
import torch
from torch.optim import Adam
from torch.utils.tensorboard import SummaryWriter
import argparse
writer = SummaryWriter()

bkbone = ['resnet18','resenet34','resenet50','resnet101']

for bbone in bkbone:

    ap = argparse.ArgumentParser()
    ap.add_argument("-b", "--backbone", required = False, default = bbone, help = "resenet18, resnet34, resnet50, resenet101 and mobilenet_v2 can be given")
    args = vars(ap.parse_args())
    epochs = 1000
    batch_size = 10
    lr = 0.005
    path = 'D:\\beantech_Data\\objtect_detection'
    # path = '/media/TBDataStudent/pankaj/beantech/object_detection'
    data= dataloader.Bar(root=path, batch_size=batch_size)

    model = md.Faster_RCNN(backbone=args['backbone']).cuda()
    model.train()

    optimizer = Adam(model.parameters(), lr = lr, weight_decay=0.0005)

    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,step_size=3,gamma=0.1)

    ### Train ###
    for epoch in range(epochs):
        loss_classifier=[]
        loss_box_reg =[]
        loss_objectness = []
        loss_rpn_box_reg = []
        Tloss = []

        for _, (img, label) in enumerate(data.train_loader):
            images = [image.cuda() for image in img ]
            for i in range(len(label)):
                label[i]['boxes']=label[i]['boxes'].cuda()
                label[i]['labels']=label[i]['labels'].cuda()
            optimizer.zero_grad()
            output = model(images, label)

            l1, l2, l3, l4 = output
            loss_classifier.append(output[l1].item())
            loss_box_reg.append(output[l2].item())
            loss_objectness.append(output[l3].item())
            loss_rpn_box_reg.append(output[l4].item())

            loss = sum(list(output.values()))
            loss.backward()
            optimizer.step()
            Tloss.append(loss.item())

        writer.add_scalar(l1, np.mean(loss_classifier), epoch)
        writer.add_scalar(l2, np.mean(loss_box_reg), epoch)
        writer.add_scalar(l3, np.mean(loss_objectness), epoch)
        writer.add_scalar(l4, np.mean(loss_rpn_box_reg), epoch)
        writer.add_scalar('Total Loss', np.mean(Tloss), epoch)

        print(f'\n\n --{l1}: {np.mean(loss_classifier)}\n --{l2}: {np.mean(loss_box_reg)}\n --{l3}: {np.mean(loss_objectness)}\n --{l4}: {np.mean(loss_rpn_box_reg)}')
        print(f'Total loss of epoch {epoch}is: {np.mean(Tloss)}')
        writer.close()

    torch.save(model.state_dict(), f'FasterRCNN_{args["backbone"]}'+'.pt')


title: how to check if i have cuda installed, i came across three methods but they give me different results

body:
I know that it might be because i dont have the toolkit installed but can someone please elaborate why smi shows cuda-10.1 while nvcc and /usr/local/cuda/version.txt  don't return anything
i want to install pytorch and use the gpu for training so should i go with cuda toolkit?

(base) e@c:~$ nvidia-smi
Wed May 13 05:03:08 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 930M        Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   52C    P0    N/A /  N/A |    616MiB /  2004MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2004      G   /usr/lib/xorg/Xorg                           194MiB |
|    0      2253      G   /usr/bin/gnome-shell                         159MiB |
|    0      2703      G   /proc/self/exe                               259MiB |
+-----------------------------------------------------------------------------+

(base) e@c:~$ nvcc --version
Command 'nvcc' not found, but can be installed with:
sudo apt install nvidia-cuda-toolkit

(base) e@c:~$ cat /usr/local/cuda/version.txt
cat: /usr/local/cuda/version.txt: No such file or directory


title: RoI cropping in Spatial Transformer Network

body:
TL;DR: How does RoI cropping method from Spatial Transformer Network works?

Reading PyTorch Spatial Transformer Network tutorial I saw the network uses a special RoI pooling I haven't seen before called RoI cropping.
Reading the docs for the F.affine_grid and F.grid_sample did not explain so much what is happening there, so I tried reading the network's paper in hope to understand as well as some blog post on Faster RCNN elaborating on the method with pictures, but still no help.
I feel every source has different details and can't get the idea of what exactly is happening there, as good as I understand the normal RoI pooling and align methods.

Right now, this is the big picture in my head:
1. As usual, map the suggested RoI coordinates to the feature map space.
2. Normalize the coordinates to the range of [-1, 1] (I guess that's for the following affine transformation).
3. Calculate (using the method in the picture below) the transformation values.
4. Now, I assume we apply the transformation to the RoI pixels?
5. And finally, I assume we do an interpolation (i.e. bilinear interpolation) to the final coordinates.

Could someone briefly explain the whole process of the RoI cropping method? I feel I might be missing something.


title: Pytorch: weights not updating

body:
I'm trying to train a model, and it doesn't work because weights aren't updating when I call the following:

self.optimizer = Adam(self.PPO.parameters(), lr=0.1, eps=epsilon)
total_loss = Variable(policy_loss + 0.5*value_loss - entropy_loss.mean() * 0.01, requires_grad=True)

self.optimizer.zero_grad()
(total_loss * 10).backward()
self.optimizer.step()

When I print the weights, they're all the same (loss isn't zero, and learning rate set to 0.1), and when I compare them (even with clone() called on each param) it always returns True. Total loss has a grad_fn attribute too... The optimizer is created in the constructor of my agent class.

My code is based on this repository:
https://github.com/andreiliphd/tennis-ppo/blob/master/agent.py

This is my agent constructor:

    def __init__(self, PPO, learning_rate, epsilon, discount_rate, entropy_coefficient, ppo_clip, gradient_clip,
                 rollout_length, tau):
        self.PPO = PPO
        self.learning_rate = learning_rate
        self.epsilon = epsilon
        self.discount_rate = discount_rate
        self.entropy_coefficient = entropy_coefficient
        self.ppo_clip = 0.2
        self.gradient_clip = 5
        self.rollout_length = rollout_length
        self.tau = tau
        self.optimizer = Adam(self.PPO.actor.parameters(), lr=0.1, eps=epsilon)
        self.device = torch.device('cpu')

This is my PPO class, which creates two networks with a forward function, and some hidden layers

class PPO(nn.Module):

    def __init__(self, state_shape, action_num, mlp_layers, device=torch.device('cpu')):
        super(PPO, self).__init__()
        self.state_shape = state_shape
        self.action_num = action_num
        self.mlp_layers = mlp_layers
        self.device = torch.device('cpu')

        layer_dims = [np.prod(self.state_shape)] + self.mlp_layers
        self.actor = PPO_Network(state_shape, action_num, layer_dims, True)
        self.actor = self.actor.to(device)
        self.critic = PPO_Network(state_shape, 1, layer_dims, False)
        self.critic = self.critic.to(device)
        self.to(device)

Any indications on why this is happening, and what I am overlooking are very welcome. :)
I can give more info or code if needed.


title: Cannot import torch problem [WinError 126] but the file is in place

body:
I have tried to import torch after I have installed it following the official site. However, I got this error. I seems like ctypes.CDLL(c:\Users\Jianr\Miniconda3\lib\site-packages\torch\lib\caffe2_nvrtc.dll) returned this error. And this file is indeed at the location specified. This is an absolute path. And the dll files at the same folder does not cause such problem. What could be the reason that this is happening and how could I fix it? Thank you!

Error Message:

Traceback (most recent call last):
File "C:/Users/Jianr/Downloads/cal_similarity.py", line 2, in <module>
    import torch
File "c:\Users\Jianr\Miniconda3\Lib\site-packages\torch\__init__.py", line 82, in <module>
    ctypes.CDLL(dll)
File "c:\Users\Jianr\Miniconda3\Lib\ctypes\__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
builtins.OSError: [WinError 126] The specified module could not be found```


title: Using PyTorch for computing derivative of a function

body:
When I execute the code below:
import torch

def g(x):
    return 4*x + 3
x=3.0
g_hat=torch.tensor(g(x), requires_grad= True)
g_hat.backward()

I get the following output:
>>>g_hat.grad
tensor(1.)

But this is not the result that I was expecting to get from my code above... what I want to do is, I want to find the value of dg/dx, at x = 3.0 (so in the example above, the correct output should be tensor(4.)).
How can I achieve this with PyTorch? or if I can't carry out this task with PyTorch, how can I do this type of task easily on Python?
Thank you,


title: Dataset Formatting Issue in face.evoLVe.PyTorch

body:
So I've been trying to train a large scale facial recognition model through the evoLVe.PyTorch library but im running into a lot of problems concerning the structure and formatting for the datasets. To be specific I will be validating the pre-trained model (pre-trained with MS-Celeb-1M) with LFW, CFP_FF, CFP_FP, AgeDB, CALFW, CPLFW, & Vggface2_FP. The backbone is IR-152, the head is ArcFace, and the loss is Focal.
I ran the train.py file and this is what it outputted:
============================================================
Overall Configurations:
{'SEED': 1337, 'DATA_ROOT': '/Volumes/storage/DeepLearning/data/CASIA-maxpy-clean', 'MODEL_ROOT': '/Volumes/storage/DeepLearning/checkpoint', 'LOG_ROOT': '/Volumes/storage/DeepLearning/log', 'BACKBONE_RESUME_ROOT': './', 'HEAD_RESUME_ROOT': './', 'BACKBONE_NAME': 'IR_SE_50', 'HEAD_NAME': 'ArcFace', 'LOSS_NAME': 'Focal', 'INPUT_SIZE': [112, 112], 'RGB_MEAN': [0.5, 0.5, 0.5], 'RGB_STD': [0.5, 0.5, 0.5], 'EMBEDDING_SIZE': 512, 'BATCH_SIZE': 512, 'DROP_LAST': True, 'LR': 0.1, 'NUM_EPOCH': 125, 'WEIGHT_DECAY': 0.0005, 'MOMENTUM': 0.9, 'STAGES': [30, 60, 90], 'DEVICE': device(type='cpu'), 'MULTI_GPU': True, 'GPU_ID': [0, 1, 2, 3], 'PIN_MEMORY': True, 'NUM_WORKERS': 0}
============================================================
Traceback (most recent call last):
  File "train.py", line 70, in <module>
    dataset_train = datasets.ImageFolder(os.path.join(DATA_ROOT, 'imgs'), train_transform)
  File "/Users/royce.moon/evoLVeTest/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 206, in __init__
    is_valid_file=is_valid_file)
  File "/Users/royce.moon/evoLVeTest/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 94, in __init__
    classes, class_to_idx = self._find_classes(self.root)
  File "/Users/royce.moon/evoLVeTest/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 121, in _find_classes
    classes = [d.name for d in os.scandir(dir) if d.is_dir()]
FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/storage/DeepLearning/data/CASIA-maxpy-clean/imgs'

Please help as I am not sure how to fix this error and get train.py to run smoothly.


title: Problem with output of neural network in a cross-entropy method attempt at solving CartPole-v0

body:
I am trying to implement the cross-entropy policy-based method to the classic CartPole-v0 environment. I am actually reformatting a working implementation of this algorithm on the MountainCarContinuous-v0, but when I try to get the agent learning, I get this error message:
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
 in
      4
      5 agent = Agent(env)
----> 6 scores = agent.learn()
      7
      8 # plot the scores

~/cross_entropy.py in learn(self, n_iterations, max_t, gamma, print_every, pop_size, elite_frac, sigma)
     83         for i_iteration in range(1, n_iterations+1): # loop over all the training iterations
     84             weights_pop = [best_weight + (sigma*np.random.randn(self.get_weights_dim())) for i in range(pop_size)] # population of the weights/policies
---> 85             rewards = np.array([self.evaluate(weights, gamma, max_t) for weights in weights_pop]) # rewards from the policies resulting from all individual weights
     86
     87             # get the best policies

~/cross_entropy.py in (.0)
     83         for i_iteration in range(1, n_iterations+1): # loop over all the training iterations
     84             weights_pop = [best_weight + (sigma*np.random.randn(self.get_weights_dim())) for i in range(pop_size)] # population of the weights/policies
---> 85             rewards = np.array([self.evaluate(weights, gamma, max_t) for weights in weights_pop]) # rewards from the policies resulting from all individual weights
     86
     87             # get the best policies

~/cross_entropy.py in evaluate(self, weights, gamma, max_t)
     56             action = self.forward(state)
     57             #action = torch.argmax(action_vals).item()
---> 58             state, reward, done, _ = self.env.step(action)
     59             episode_return += reward * math.pow(gamma, t)
     60             if done:

/gym/wrappers/time_limit.py in step(self, action)
     14     def step(self, action):
     15         assert self._elapsed_steps is not None, "Cannot call env.step() before calling reset()"
---> 16         observation, reward, done, info = self.env.step(action)
     17         self._elapsed_steps += 1
     18         if self._elapsed_steps >= self._max_episode_steps:

/gym/envs/classic_control/cartpole.py in step(self, action)
    102     def step(self, action):
    103         err_msg = "%r (%s) invalid" % (action, type(action))
--> 104         assert self.action_space.contains(action), err_msg
    105
    106         x, x_dot, theta, theta_dot = self.state

AssertionError: tensor([ 0.3987,  0.6013]) () invalid

I found this is because the MountainCarContinuous-v0 environment has an action_space of type Box(2) whereas CartPole-v0 is Discrete(2), meaning that I only want an integer as action selection.
I have tried working around this notion by applying a softmax activation function and then took the index of the higher value as the action.
action_vals = self.forward(state)
action = torch.argmax(action_vals).item()

This gets rid of the error but when I train the agent, it seems to learn incredibly fast which is kind of an indicator that something is wrong. This is my full agent class:
class Agent(nn.Module):
    def __init__(self, env, h_size=16):
        super().__init__()
        self.env = env
        # state, hidden layer, action sizes
        self.s_size = env.observation_space.shape[0]
        self.h_size = h_size
        self.a_size = env.action_space.n
        # define layers
        self.fc1 = nn.Linear(self.s_size, self.h_size)
        self.fc2 = nn.Linear(self.h_size, self.a_size)

        self.device = torch.device('cpu')

    def set_weights(self, weights):
        s_size = self.s_size
        h_size = self.h_size
        a_size = self.a_size
        # separate the weights for each layer
        fc1_end = (s_size*h_size)+h_size
        fc1_W = torch.from_numpy(weights[:s_size*h_size].reshape(s_size, h_size))
        fc1_b = torch.from_numpy(weights[s_size*h_size:fc1_end])
        fc2_W = torch.from_numpy(weights[fc1_end:fc1_end+(h_size*a_size)].reshape(h_size, a_size))
        fc2_b = torch.from_numpy(weights[fc1_end+(h_size*a_size):])
        # set the weights for each layer
        self.fc1.weight.data.copy_(fc1_W.view_as(self.fc1.weight.data))
        self.fc1.bias.data.copy_(fc1_b.view_as(self.fc1.bias.data))
        self.fc2.weight.data.copy_(fc2_W.view_as(self.fc2.weight.data))
        self.fc2.bias.data.copy_(fc2_b.view_as(self.fc2.bias.data))

    def get_weights_dim(self):
        return (self.s_size+1)*self.h_size + (self.h_size+1)*self.a_size

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.softmax(self.fc2(x))
        return x

    def evaluate(self, weights, gamma=1.0, max_t=5000):
        self.set_weights(weights)
        episode_return = 0.0
        state = self.env.reset()
        for t in range(max_t):
            state = torch.from_numpy(state).float().to(self.device)
            action_vals = self.forward(state)
            action = torch.argmax(action_vals).item()
            state, reward, done, _ = self.env.step(action)
            episode_return += reward * math.pow(gamma, t)
            if done:
                break
        return episode_return

    def learn(self, n_iterations=500, max_t=1000, gamma=1.0, print_every=10, pop_size=50, elite_frac=0.2, sigma=0.5):
        """PyTorch implementation of the cross-entropy method.

        Params
        ======
            n_iterations (int): maximum number of training iterations
            max_t (int): maximum number of timesteps per episode
            gamma (float): discount rate
            print_every (int): how often to print average score (over last 100 episodes)
            pop_size (int): size of population at each iteration
            elite_frac (float): percentage of top performers to use in update
            sigma (float): standard deviation of additive noise
        """
        n_elite=int(pop_size*elite_frac) # number of elite policies from the population

        scores_deque = deque(maxlen=100) # list of the past 100 scores
        scores = [] # list of all the scores
        best_weight = sigma*np.random.randn(self.get_weights_dim()) # initialize the first best weight randomly

        for i_iteration in range(1, n_iterations+1): # loop over all the training iterations
            weights_pop = [best_weight + (sigma*np.random.randn(self.get_weights_dim())) for i in range(pop_size)] # population of the weights/policies
            rewards = np.array([self.evaluate(weights, gamma, max_t) for weights in weights_pop]) # rewards from the policies resulting from all individual weights

            # get the best policies
            ##
            elite_idxs = rewards.argsort()[-n_elite:]
            elite_weights = [weights_pop[i] for i in elite_idxs]
            ##

            best_weight = np.array(elite_weights).mean(axis=0) # take the average of the best weights

            reward = self.evaluate(best_weight, gamma=1.0) # evaluate this new policy
            scores_deque.append(reward) # append the reward
            scores.append(reward) # also append the reward

            torch.save(self.state_dict(), 'checkpoint.pth') # save the agent

            if i_iteration % print_every == 0: # print every 100 steps
                print('Episode {}\tAverage Score: {:.2f}'.format(i_iteration, np.mean(scores_deque)))

            if np.mean(scores_deque)>=195.0: # print if environment is solved
                print('\nEnvironment solved in {:d} iterations!\tAverage Score: {:.2f}'.format(i_iteration-100, np.mean(scores_deque)))
                break
        return scores

If anyone has an idea on how to get the agent training properly, please give me any suggestions.


title: Fill regions from contour based on cumsum

body:
I got an image for scene classification:

And using cumsum, I want to segment the three parts of it.
I take this simple operation in pytorch (can use tensorflow of course too, or python)


title: QValues explosion DQN

body:
I am trying to follow the tutorial of pytorch about QLearning. I am keeping the core but try to implement it a little bit differently. However, I stumble upon issues. In the best case the algorithm doesn't seem to solve the environment, and in the worst case an explosion of the predicted QValues occurs. I cannot find the root reason for that
I tried to tune most of the hyperparameters, lower the learning rate, and I checked my code several times to see if there is a difference between my code and the code on the pytorch website.
Does someone know what can cause this behavior? Or see a mistake in the code? I checked most of the post from stack overflow, most of them speak about the learning rates or the target network. There is a target network in my code, and I tried to lower the LR, but it does'nt seem to solve the issue unfortunately
import gym
import torch
import random
import numpy as np
from collections import deque
from copy import deepcopy
import torch.nn as nn
import torch.optim as optim
from itertools import count
import torch.nn.functional as F
from collections import namedtuple

GAMMA = 0.99
EPSILON_END = 0.05
EPSILON_START = 0.9
EPSILON_DECAY = 500
LEARNING_RATE = 0.0001

ACTION_SPACE = 4
OBSERVATION_SPACE = 8

BATCH_SIZE = 32
MEMORY_CAPACITY = 10000
TARGET_NETWORK_UPDATE_FREQUENCY = 2

Experience = namedtuple('Experience', ['state', 'action', 'reward', 'next_state'])

class Network(nn.Module):

    def __init__(self, input_features, output_features):
        super().__init__()
        self.fc_1 = nn.Linear(input_features, 100)
        self.bn_1 = nn.LayerNorm(100)
        self.fc_2 = nn.Linear(100, 100)
        self.bn_2 = nn.LayerNorm(100)
        self.fc_3 = nn.Linear(100, output_features)

    def forward(self, x):
        x = F.relu(self.bn_1(self.fc_1(x)))
        x = F.relu(self.bn_2(self.fc_2(x)))
        return self.fc_3(x)

class Memory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, experience):
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = experience
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)

class Agent:

    def __init__(self, policy_network, action_space):
        self.policy_network = policy_network
        self.actions = list(range(action_space))

    def __call__(self, observation, step_id=None):
        if np.random.random() > get_epsilon(step_id):
            with torch.no_grad():
                predictions = self.policy_network(torch.tensor(observation)).detach().numpy()
                return np.argmax(predictions)
        else:
            return np.random.choice(self.actions)

class QLearning:

    def __init__(self, agent: Agent, target_network: nn.Module, optimizer, memory_capacity, batch_size):
        self.step = 0
        self.agent = agent
        self.target_network = target_network
        self.optimizer = optimizer
        self.batch_size = batch_size
        self.memory = Memory(memory_capacity)

    def train(self, num_episodes):
        env = gym.make('LunarLander-v2')
        recent_scores = deque(maxlen=100)
        for episode_id in range(num_episodes):
            observation = env.reset()
            score = 0
            for t in count():
                action = self.agent(observation, self.step)
                observation_, reward, done, info = env.step(action.item())

                observation_ = None if done else observation_
                experience = self.preprocess_experience(observation, action, reward, observation_)
                self.memory.push(experience)

                q_values = self.learn()
                self.step += 1
                score += reward

                if done is True:
                    break

            recent_scores.append(score)

            if episode_id % TARGET_NETWORK_UPDATE_FREQUENCY == 0:
                self.update_target_network()

            if episode_id % 10 == 0:
                print(episode_id, np.mean(recent_scores), t)  # TODO Tensorboard
                print('sample of the state_action_values :', list(q_values))

    def learn(self):
        if len(self.memory) < self.batch_size:
            return
        experiences = self.memory.sample(self.batch_size)
        batch = Experience(*zip(*experiences))

        states = torch.cat(batch.state)
        actions = torch.cat(batch.action)
        rewards = torch.cat(batch.reward)
        state_action_values = self.agent.policy_network(states).gather(1, actions)

        non_terminal_mask = torch.tensor(tuple(map(lambda s: s is not None, batch.next_state)), dtype=torch.bool)
        non_terminal_next_states = torch.cat([s for s in batch.next_state if s is not None])

        next_state_values = torch.zeros(BATCH_SIZE)
        next_state_values[non_terminal_mask] = self.target_network(non_terminal_next_states).max(1)[0].detach()

        expected_state_action_values = (next_state_values * GAMMA) + rewards
        expected_state_action_values = expected_state_action_values.unsqueeze(1)

        loss = F.smooth_l1_loss(state_action_values, expected_state_action_values)
        self.optimizer.zero_grad()
        loss.backward()

        for param in self.agent.policy_network.parameters():
            param.grad.data.clamp_(-1, 1)
        self.optimizer.step()

        return state_action_values.detach().tolist()[:3] # This is just to monitor if there is an issue with the values predicted

    def update_target_network(self):
        self.target_network.load_state_dict(self.agent.policy_network.state_dict())

    @staticmethod
    def preprocess_experience(state, action, reward, next_state):
        state = torch.tensor(state).unsqueeze(0)
        next_state = torch.tensor(next_state).unsqueeze(0) if next_state is not None else None
        action = torch.tensor(action, dtype=torch.int64).view(1, 1)
        reward = torch.tensor(reward, dtype=torch.float32).view(1)
        return Experience(state, action, reward, next_state)

def get_networks_and_optimizer(learning_rate, observation_space, action_space):
    policy_network = Network(observation_space, action_space)
    target_network = deepcopy(policy_network)
    target_network.eval()
    policy_network_optimizer = optim.RMSprop(policy_network.parameters(), learning_rate)
    return policy_network, target_network, policy_network_optimizer

def get_epsilon(step):
    if step is None:
        return EPSILON_END
    return EPSILON_END + (EPSILON_START - EPSILON_END) * np.exp(-1. * step / EPSILON_DECAY)

if __name__ == '__main__':
    policy_network, target_network, policy_network_optimizer = get_networks_and_optimizer(LEARNING_RATE, OBSERVATION_SPACE, ACTION_SPACE)
    agent = Agent(policy_network, ACTION_SPACE)
    trainer = QLearning(agent, target_network, policy_network_optimizer, MEMORY_CAPACITY, BATCH_SIZE)
    trainer.train(10000)


I test my code, but someting wrong. I use the pytorch to load data and test ICP and i have used "cuda()" to add some variables to the GPU. However, the pycharm shows the error information"RuntimeError: Expected object of device type cuda but got device type cpu for argument #0 'result' in call to _th_bmm_out" . I really don't know how to slove this problem.
Here is the ICP code.
import numpy as np
from sklearn.neighbors import NearestNeighbors

def best_fit_transform(A, B):
    '''
    Calculates the least-squares best-fit transform that maps corresponding points A to B in m spatial dimensions
    Input:
      A: Nxm numpy array of corresponding points
      B: Nxm numpy array of corresponding points
    Returns:
      T: (m+1)x(m+1) homogeneous transformation matrix that maps A on to B
      R: mxm rotation matrix
      t: mx1 translation vector
    '''

    assert A.shape == B.shape  #条件为true正常执行，条件为false触发异常

    # get number of dimensions
    m = A.shape[1]

    # translate points to their centroids
    #计算点云质心
    centroid_A = np.mean(A, axis=0)
    centroid_B = np.mean(B, axis=0)
    #源、目标点云计算去质心
    AA = A - centroid_A
    BB = B - centroid_B

    # rotation matrix
    H = np.dot(AA.T, BB)
    #SVD分解H协方差矩阵
    U, S, Vt = np.linalg.svd(H)
    R = np.dot(Vt.T, U.T)

    # special reflection case
    if np.linalg.det(R) < 0:
       Vt[m-1,:] *= -1
       R = np.dot(Vt.T, U.T)

    # translation
    #t:3X1
    t = centroid_B.T - np.dot(R,centroid_A.T)

    # homogeneous transformation
    #T为m+1维的对角矩阵
    T = np.identity(m+1)
    T[:m, :m] = R
    T[:m, m] = t

    return T, R, t

def nearest_neighbor(src, dst):
    '''
    Find the nearest (Euclidean) neighbor in dst for each point in src
    Input:
        src: Nxm array of points
        dst: Nxm array of points
    Output:
        distances: Euclidean distances of the nearest neighbor
        indices: dst indices of the nearest neighbor
    '''

    assert src.shape == dst.shape

    neigh = NearestNeighbors(n_neighbors=1)
    neigh.fit(dst)
    distances, indices = neigh.kneighbors(src, return_distance=True)
    return distances.ravel(), indices.ravel()

def icp(A, B, init_pose=None, max_iterations=20, tolerance=0.001):
    '''
    The Iterative Closest Point method: finds best-fit transform that maps points A on to points B
    Input:
        A: Nxm numpy array of source mD points
        B: Nxm numpy array of destination mD point
        init_pose: (m+1)x(m+1) homogeneous transformation
        max_iterations: exit algorithm after max_iterations
        tolerance: convergence criteria
    Output:
        T: final homogeneous transformation that maps A on to B
        distances: Euclidean distances (errors) of the nearest neighbor
        i: number of iterations to converge
    '''

    assert A.shape == B.shape

    # get number of dimensions
    m = A.shape[1]

    # make points homogeneous, copy them to maintain the originals
    src = np.ones((m+1,A.shape[0]))
    # print(type(src))
    dst = np.ones((m+1,B.shape[0]))
    # print(dst.shape)
    src[:m,:] = np.copy(A.T)
    dst[:m,:] = np.copy(B.T)

    # apply the initial pose estimation
    if init_pose is not None:
        src = np.dot(init_pose, src)

    prev_error = 0

    for i in range(max_iterations):
        # find the nearest neighbors between the current source and destination points
        distances, indices = nearest_neighbor(src[:m,:].T, dst[:m,:].T)

        # compute the transformation between the current source and nearest destination points
        T,_,_ = best_fit_transform(src[:m,:].T, dst[:m,indices].T)

        # update the current source
        src = np.dot(T, src)

        # check error
        mean_error = np.mean(distances)
        if np.abs(prev_error - mean_error) < tolerance:
            break
        prev_error = mean_error

    # calculate final transformation
    T,_,_ = best_fit_transform(A, src[:m,:].T)

    return T, distances, i

Here is the testICP code.
import numpy as np
import time
from ICP import icp
from data import ModelNet40
from tqdm import tqdm
from util import transform_point_cloud, npmat2euler
from torch.utils.data import DataLoader
from tensorboardX import SummaryWriter
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.optim.lr_scheduler import MultiStepLR

def test_one_epoch(test_loader):

    mse_ab = 0
    mae_ab = 0
    mse_ba = 0
    mae_ba = 0

    total_loss = 0
    total_cycle_loss = 0
    num_examples = 0
    rotations_ab = []
    translations_ab = []
    rotations_ab_pred = []
    translations_ab_pred = []

    rotations_ba = []
    translations_ba = []
    rotations_ba_pred = []
    translations_ba_pred = []

    eulers_ab = []
    eulers_ba = []

    for src, target, rotation_ab, translation_ab, rotation_ba, translation_ba, euler_ab, euler_ba in tqdm(test_loader):
        src = src.cuda()
        src1 = src.cpu().numpy()
        src1 = np.reshape(src1,(-1,1024))
        src1 = src1.T
        # print(src.shape)
        target = target.cuda()
        target1 = target.cpu().numpy()
        target1 = np.reshape(target1,(-1,1024))
        target1 = target1.T
        rotation_ab = rotation_ab.cuda()
        translation_ab = translation_ab.cuda()
        # rotation_ba = rotation_ba.cuda()
        # translation_ba = translation_ba.cuda()

        batch_size = 1
        num_examples += batch_size
        # rotation_ab_pred, translation_ab_pred, rotation_ba_pred, translation_ba_pred = net(src, target)

        T, distances, iterations= icp.icp(src1, target1, tolerance=0.000001)
        rotation_ab_pred = T[0:3,0:3]
        rotation_ab_pred = np.reshape(rotation_ab_pred,(1,3,3))
        rotation_ab_pred = torch.from_numpy(rotation_ab_pred)
        rotation_ab_pred = rotation_ab_pred.cuda()
        rotation_ab_pred = torch.tensor(rotation_ab_pred, dtype=torch.float32)
        translation_ab_pred = T[0:3,3]
        translation_ab_pred = np.reshape(translation_ab_pred,(1,3))
        translation_ab_pred = torch.from_numpy(translation_ab_pred)
        translation_ab_pred = translation_ab_pred.cuda()
        translation_ab_pred = torch.tensor(translation_ab_pred,dtype=torch.float32)

        ## save rotation and translation
        rotations_ab.append(rotation_ab.detach().cpu().numpy())
        translations_ab.append(translation_ab.detach().cpu().numpy())
        rotations_ab_pred.append(rotation_ab_pred.detach().cpu().numpy())
        translations_ab_pred.append(translation_ab_pred.detach().cpu().numpy())
        eulers_ab.append(euler_ab.numpy())
        ##
        # rotations_ba.append(rotation_ba.detach().cpu().numpy())
        # translations_ba.append(translation_ba.detach().cpu().numpy())
        # rotations_ba_pred.append(rotation_ba_pred.detach().cpu().numpy())
        # translations_ba_pred.append(translation_ba_pred.detach().cpu().numpy())
        # eulers_ba.append(euler_ba.numpy())

        transformed_src = transform_point_cloud(src, rotation_ab_pred, translation_ab_pred)
        # transformed_src = transformed_src.cuda()

        # transformed_target = transform_point_cloud(target, rotation_ba_pred, translation_ba_pred)

        ###########################
        identity = torch.eye(3).cuda().unsqueeze(0).repeat(batch_size, 1, 1)
        loss = F.mse_loss(torch.matmul(rotation_ab_pred.transpose(2, 1), rotation_ab), identity) \
               + F.mse_loss(translation_ab_pred, translation_ab)
        # if args.cycle:
        #     rotation_loss = F.mse_loss(torch.matmul(rotation_ba_pred, rotation_ab_pred), identity.clone())
        #     translation_loss = torch.mean((torch.matmul(rotation_ba_pred.transpose(2, 1),
        #                                                 translation_ab_pred.view(batch_size, 3, 1)).view(batch_size, 3)
        #                                    + translation_ba_pred) ** 2, dim=[0, 1])
        #     cycle_loss = rotation_loss + translation_loss
        #
        #     loss = loss + cycle_loss * 0.1

        total_loss += loss.item() * batch_size

        # if args.cycle:
        #     total_cycle_loss = total_cycle_loss + cycle_loss.item() * 0.1 * batch_size

        mse_ab += torch.mean((transformed_src - target) ** 2, dim=[0, 1, 2]).item() * batch_size
        mae_ab += torch.mean(torch.abs(transformed_src - target), dim=[0, 1, 2]).item() * batch_size

        # mse_ba += torch.mean((transformed_target - src) ** 2, dim=[0, 1, 2]).item() * batch_size
        # mae_ba += torch.mean(torch.abs(transformed_target - src), dim=[0, 1, 2]).item() * batch_size

    rotations_ab = np.concatenate(rotations_ab, axis=0)
    translations_ab = np.concatenate(translations_ab, axis=0)
    rotations_ab_pred = np.concatenate(rotations_ab_pred, axis=0)
    translations_ab_pred = np.concatenate(translations_ab_pred, axis=0)

    # rotations_ba = np.concatenate(rotations_ba, axis=0)
    # translations_ba = np.concatenate(translations_ba, axis=0)
    # rotations_ba_pred = np.concatenate(rotations_ba_pred, axis=0)
    # translations_ba_pred = np.concatenate(translations_ba_pred, axis=0)

    eulers_ab = np.concatenate(eulers_ab, axis=0)
    # eulers_ba = np.concatenate(eulers_ba, axis=0)

    return total_loss * 1.0 / num_examples, \
           mse_ab * 1.0 / num_examples, mae_ab * 1.0 / num_examples, \
           rotations_ab, \
           translations_ab, rotations_ab_pred, translations_ab_pred, eulers_ab

def test(test_loader):

    test_loss,  \
    test_mse_ab, test_mae_ab, test_rotations_ab, test_translations_ab, \
    test_rotations_ab_pred, \
    test_translations_ab_pred, \
    test_eulers_ab = test_one_epoch(test_loader)
    test_rmse_ab = np.sqrt(test_mse_ab)

    test_rotations_ab_pred_euler = npmat2euler(test_rotations_ab_pred)
    test_r_mse_ab = np.mean((test_rotations_ab_pred_euler - np.degrees(test_eulers_ab)) ** 2)
    test_r_rmse_ab = np.sqrt(test_r_mse_ab)
    test_r_mae_ab = np.mean(np.abs(test_rotations_ab_pred_euler - np.degrees(test_eulers_ab)))
    test_t_mse_ab = np.mean((test_translations_ab - test_translations_ab_pred) ** 2)
    test_t_rmse_ab = np.sqrt(test_t_mse_ab)
    test_t_mae_ab = np.mean(np.abs(test_translations_ab - test_translations_ab_pred))

    # test_rotations_ba_pred_euler = npmat2euler(test_rotations_ba_pred, 'xyz')
    # test_r_mse_ba = np.mean((test_rotations_ba_pred_euler - np.degrees(test_eulers_ba)) ** 2)
    # test_r_rmse_ba = np.sqrt(test_r_mse_ba)
    # test_r_mae_ba = np.mean(np.abs(test_rotations_ba_pred_euler - np.degrees(test_eulers_ba)))
    # test_t_mse_ba = np.mean((test_translations_ba - test_translations_ba_pred) ** 2)
    # test_t_rmse_ba = np.sqrt(test_t_mse_ba)
    # test_t_mae_ba = np.mean(np.abs(test_translations_ba - test_translations_ba_pred))

    print('==FINAL TEST==')
    print('A--------->B')
    print('EPOCH:: %d, Loss: %f, MSE: %f, RMSE: %f, MAE: %f, rot_MSE: %f, rot_RMSE: %f, '
                  'rot_MAE: %f, trans_MSE: %f, trans_RMSE: %f, trans_MAE: %f'
                  % (-1, test_loss, test_mse_ab, test_rmse_ab, test_mae_ab,test_r_mse_ab, test_r_rmse_ab,
                     test_r_mae_ab, test_t_mse_ab, test_t_rmse_ab, test_t_mae_ab))
    # textio.cprint('B--------->A')
    # textio.cprint('EPOCH:: %d, Loss: %f, MSE: %f, RMSE: %f, MAE: %f, rot_MSE: %f, rot_RMSE: %f, '
    #               'rot_MAE: %f, trans_MSE: %f, trans_RMSE: %f, trans_MAE: %f'
    #               % (-1, test_loss, test_mse_ba, test_rmse_ba, test_mae_ba, test_r_mse_ba, test_r_rmse_ba,
    #                  test_r_mae_ba, test_t_mse_ba, test_t_rmse_ba, test_t_mae_ba))

def main():

    test_loader = DataLoader(
            ModelNet40(num_points=1024, partition='test', gaussian_noise=False,
                       unseen=False, factor=4),
            batch_size=1, shuffle=False, drop_last=False)
    test(test_loader)

    print('FINISH')

if __name__ == '__main__':
    main()

Here is the data.py.
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os
import sys
import glob
import h5py
import numpy as np
from scipy.spatial.transform import Rotation
from torch.utils.data import Dataset

# Part of the code is referred from: https://github.com/charlesq34/pointnet

def download():
    BASE_DIR = os.path.dirname(os.path.abspath(__file__))
    DATA_DIR = os.path.join(BASE_DIR, 'data')
    if not os.path.exists(DATA_DIR):
        os.mkdir(DATA_DIR)
    if not os.path.exists(os.path.join(DATA_DIR, 'modelnet40_ply_hdf5_2048')):
        www = 'https://shapenet.cs.stanford.edu/media/modelnet40_ply_hdf5_2048.zip'
        zipfile = os.path.basename(www)
        os.system('wget %s; unzip %s' % (www, zipfile))
        os.system('mv %s %s' % (zipfile[:-4], DATA_DIR))
        os.system('rm %s' % (zipfile))

def load_data(partition):
    download()
    BASE_DIR = os.path.dirname(os.path.abspath(__file__))
    DATA_DIR = os.path.join(BASE_DIR, 'data')
    all_data = []
    all_label = []
    for h5_name in glob.glob(os.path.join(DATA_DIR, 'modelnet40_ply_hdf5_2048', 'ply_data_%s*.h5' % partition)):
        f = h5py.File(h5_name)
        data = f['data'][:].astype('float32')
        label = f['label'][:].astype('int64')
        f.close()
        all_data.append(data)
        all_label.append(label)
    all_data = np.concatenate(all_data, axis=0)
    all_label = np.concatenate(all_label, axis=0)
    return all_data, all_label

def translate_pointcloud(pointcloud):
    xyz1 = np.random.uniform(low=2. / 3., high=3. / 2., size=[3])
    xyz2 = np.random.uniform(low=-0.2, high=0.2, size=[3])

    translated_pointcloud = np.add(np.multiply(pointcloud, xyz1), xyz2).astype('float32')
    return translated_pointcloud

def jitter_pointcloud(pointcloud, sigma=0.01, clip=0.05):
    N, C = pointcloud.shape
    pointcloud += np.clip(sigma * np.random.randn(N, C), -1 * clip, clip)
    return pointcloud

class ModelNet40(Dataset):
    def __init__(self, num_points, partition='train', gaussian_noise=False, unseen=False, factor=4):
        self.data, self.label = load_data(partition)
        self.num_points = num_points
        self.partition = partition
        self.gaussian_noise = gaussian_noise
        self.unseen = unseen
        self.label = self.label.squeeze()
        self.factor = factor
        if self.unseen:
            ######## simulate testing on first 20 categories while training on last 20 categories
            if self.partition == 'test':
                self.data = self.data[self.label>=20]
                self.label = self.label[self.label>=20]
            elif self.partition == 'train':
                self.data = self.data[self.label<20]
                self.label = self.label[self.label<20]

    def __getitem__(self, item):
        pointcloud = self.data[item][:self.num_points]
        if self.gaussian_noise:
            pointcloud = jitter_pointcloud(pointcloud)
        if self.partition != 'train':
            np.random.seed(item)
        anglex = np.random.uniform() * np.pi / self.factor
        angley = np.random.uniform() * np.pi / self.factor
        anglez = np.random.uniform() * np.pi / self.factor

        cosx = np.cos(anglex)
        cosy = np.cos(angley)
        cosz = np.cos(anglez)
        sinx = np.sin(anglex)
        siny = np.sin(angley)
        sinz = np.sin(anglez)
        Rx = np.array([[1, 0, 0],
                        [0, cosx, -sinx],
                        [0, sinx, cosx]])
        Ry = np.array([[cosy, 0, siny],
                        [0, 1, 0],
                        [-siny, 0, cosy]])
        Rz = np.array([[cosz, -sinz, 0],
                        [sinz, cosz, 0],
                        [0, 0, 1]])
        R_ab = Rx.dot(Ry).dot(Rz)
        R_ba = R_ab.T
        translation_ab = np.array([np.random.uniform(-0.5, 0.5), np.random.uniform(-0.5, 0.5),
                                   np.random.uniform(-0.5, 0.5)])
        translation_ba = -R_ba.dot(translation_ab)

        pointcloud1 = pointcloud.T

        rotation_ab = Rotation.from_euler('zyx', [anglez, angley, anglex])
        pointcloud2 = rotation_ab.apply(pointcloud1.T).T + np.expand_dims(translation_ab, axis=1)

        euler_ab = np.asarray([anglez, angley, anglex])
        euler_ba = -euler_ab[::-1]

        pointcloud1 = np.random.permutation(pointcloud1.T).T
        pointcloud2 = np.random.permutation(pointcloud2.T).T

        return pointcloud1.astype('float32'), pointcloud2.astype('float32'), R_ab.astype('float32'), \
               translation_ab.astype('float32'), R_ba.astype('float32'), translation_ba.astype('float32'), \
               euler_ab.astype('float32'), euler_ba.astype('float32')

    def __len__(self):
        return self.data.shape[0]

if __name__ == '__main__':
    train = ModelNet40(1024)
    test = ModelNet40(1024, 'test')
    for data in train:
        print(len(data))
        break

Here is the util.py.
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from __future__ import print_function
import os
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
from scipy.spatial.transform import Rotation

# Part of the code is referred from: https://github.com/ClementPinard/SfmLearner-Pytorch/blob/master/inverse_warp.py

def quat2mat(quat):
    x, y, z, w = quat[:, 0], quat[:, 1], quat[:, 2], quat[:, 3]

    B = quat.size(0)

    w2, x2, y2, z2 = w.pow(2), x.pow(2), y.pow(2), z.pow(2)
    wx, wy, wz = w*x, w*y, w*z
    xy, xz, yz = x*y, x*z, y*z

    rotMat = torch.stack([w2 + x2 - y2 - z2, 2*xy - 2*wz, 2*wy + 2*xz,
                          2*wz + 2*xy, w2 - x2 + y2 - z2, 2*yz - 2*wx,
                          2*xz - 2*wy, 2*wx + 2*yz, w2 - x2 - y2 + z2], dim=1).reshape(B, 3, 3)
    return rotMat

def transform_point_cloud(point_cloud, rotation, translation):
    if len(rotation.size()) == 2:
        rot_mat = quat2mat(rotation)
    else:
        rot_mat = rotation
    return torch.matmul(rot_mat, point_cloud) + translation.unsqueeze(2)

def npmat2euler(mats, seq='zyx'):
    eulers = []
    for i in range(mats.shape[0]):
        r = Rotation.from_dcm(mats[i])
        eulers.append(r.as_euler(seq, degrees=True))
    return np.asarray(eulers, dtype='float32')

When i run the testICP.py, the error information can be listed as followed.
D:\Envs\DCP\Scripts\python.exe D:/pyproject/dcp-master/ICP/testICP.py
D:\pyproject\dcp-master\data.py:51: H5pyDeprecationWarning: The default file mode will change to 'r' (read-only) in h5py 3.0. To suppress this warning, pass the mode you need to h5py.File(), or set the global default h5.get_config().default_file_mode, or set the environment variable H5PY_DEFAULT_READONLY=1. Available modes are: 'r', 'r+', 'w', 'w-'/'x', 'a'. See the docs for details.
  f = h5py.File(h5_name)
  0%|          | 0/2468 [00:00<?, ?it/s]D:/pyproject/dcp-master/ICP/testICP.py:62: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  rotation_ab_pred = torch.tensor(rotation_ab_pred, dtype=torch.float32)
D:/pyproject/dcp-master/ICP/testICP.py:67: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  translation_ab_pred = torch.tensor(translation_ab_pred,dtype=torch.float32)
  0%|          | 0/2468 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "D:/pyproject/dcp-master/ICP/testICP.py", line 180, in <module>
    main()
  File "D:/pyproject/dcp-master/ICP/testICP.py", line 175, in main
    test(test_loader)
  File "D:/pyproject/dcp-master/ICP/testICP.py", line 138, in test
    test_eulers_ab = test_one_epoch(test_loader)
  File "D:/pyproject/dcp-master/ICP/testICP.py", line 83, in test_one_epoch
    transformed_src = transform_point_cloud(src, rotation_ab_pred, translation_ab_pred)
  File "D:\pyproject\dcp-master\util.py", line 38, in transform_point_cloud
    a = torch.matmul(rot_mat, point_cloud) + translation.unsqueeze(2)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #0 'result' in call to _th_bmm_out

Process finished with exit code 1


title: Calculating the memory usage of RoBERTa?

body:
I'm currently training Roberta for sentiment classification and having a fixed mini-batch size is killing my performance. To the extent that running the stuff I'm running will take 8-9 days to cover the training set once. Is there a general equation for calculating the memory usage of RoBERTa given n_inputs, input_length and input embedding size? If I have this I should be able to efficiently mini-batch on the fly (it has to be done during runtime due to other libraries that I'm using).
(If it helps in anyway I'm using PyTorch to do this)


title: CUDA out of memory. Tried to allocate 3.82 GiB (GPU 0; 14.73 GiB total capacity; 12.46 GiB already allocated; 7.88 MiB free; 13.79 GiB)

body:
epochs = 20

for epoch in range(0, epochs + 1):
    # Training
    if epoch > 0:  # test untrained net first
        model.train()
        train_loss = 0
        for x in train_loader:
            print(x.shape)
            x = x.to(device)
            # ===================forward=====================
            x_hat, mu, logvar = model(x)
            loss = loss_function(x_hat, x, mu, logvar)
            train_loss += loss.item()
            # ===================backward====================
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        # ===================log========================
        print(f'====> Epoch: {epoch} Average loss: {train_loss / len(train_loader.dataset):.4f}')

    # Testing

    with torch.no_grad():
        model.eval()
        test_loss = 0
        for x in test_loader:
            x = x.to(device)
            # ===================forward=====================
            x_hat, mu, logvar = model(x)
            test_loss += loss_function(x_hat, x, mu, logvar).item()
            # =====================log=======================
            means.append(mu.detach())
            logvars.append(logvar.detach())
    # ===================log========================
    test_loss /= len(test_loader.dataset)
    print(f'====> Test set loss: {test_loss:.4f}')
    display_images(x, x_hat, 1, f'Epoch {epoch}')

I am currently running the above code in a Colab session. At the line x_hat, mu, logvar = model(x) I have the error : CUDA out of memory. Tried to allocate 3.82 GiB (GPU 0; 14.73 GiB total capacity; 12.46 GiB already allocated; 7.88 MiB free; 13.79 GiB reserved in total by PyTorch)
model is a simple convolutional neural network model. It is doubtful the problem come from that model.
How can I deal with that kind of problem?


title: How to store wrong predictions during evaluation on the CNN

body:
During evaluation, I want to store unique ids that are wrongly predicted to do some more processing.
It is a multiclass prediction problem
Here is the code during the evaluation:
 outputs = model(imgs)
 loss = criterion(outputs, targets)  # Prediction error

 val_loss += loss.item()
 predicted = torch.argmax(outputs, dim=1)
 t_predicted +=predicted.cpu().tolist()

 total += targets.size(0)
 good_answers = (predicted == targets)
 correct += good_answers.sum().item()

Knowing that ids is a list of the ids of the images,  When I try to get the ids that are wrong:
wrong_ids += ids[~(good_answers.to('cpu'))]

I get this error:
add(): argument 'other' (position 1) must be Tensor


title: Torch gather middle dimension

body:
Let a be a (n, d, l) tensor. Let indices be a (n, 1) tensor, containing indices. I want to gather from a in the middle dimension tensors from indices given by indices. The resulting tensor would therefore be of shape (n, l).
n = 3
d = 2
l = 3

a = tensor([[[ 0,  1,  2],
             [ 3,  4,  5]],

            [[ 6,  7,  8],
             [ 9, 10, 11]],

            [[12, 13, 14],
             [15, 16, 17]]])

indices = tensor([[0],
                  [1],
                  [0]])

# Shape of result is (n, l)
result = tensor([[ 0,  1,  2],  # a[0, 0, :] since indices[0] == 0

                 [ 9, 10, 11],  # a[1, 1, :] since indices[1] == 1

                 [12, 13, 14]]) # a[2, 0, :] since indices[2] == 0

This is indeed similar to a.gather(1, indices), but gather won't work since indices does not have the same shape as a. How can I use gather in this setting? Or what should I use?


title: How to tranform the following TensorFlow code to PyTorch?

body:
I was searching for loss functions for a GAN model and then this one came out:
gd_loss = tf.reduce_sum(tf.square(tf.abs(dx_real) - tf.abs(dx_fake))) + \
              tf.reduce_sum(tf.square(tf.abs(dy_real) - tf.abs(dy_fake))) + \
              tf.reduce_sum(tf.square(tf.abs(dz_real) - tf.abs(dz_fake)))

but I would like to convert the following to PyTorch because I'm using PyTorch tensors:
dx_real = t_target_image[:, 1:, :, :, :] - t_target_image[:, :-1, :, :, :]

where t_target_image are tensors in TensorFlow.
How can I do that?


title: IndexError: Input hidden_x1.0.weight is undefined! error with ONNX

body: A tool I am using appears to use onnx.optimizer.optimize() and as a  result I receive an error that my first weight is undefined "IndexError: Input hidden_x1.0.weight is undefined!". Does anyone know what causes this? The model was converted originally from PyTorch to ONNX.


title: Recommended way to serve PyTorch predictions with Google AI Platform

body:
Reading through Google's documentation, it seems there are two different ways of serving predictions form PyTorch models.

Using custom prediction routines
Creating a custom container and using torchserve

Although option 1 seems more popular from reading through online blogs such as this one, it seems weird to not use option 2 since the documentation explicitly mentions PyTorch.
Can anyone provide why one might choose one option over another?


title: Do pytorch detect overflow or underflow?

body:
Because deep neural net and RNN involve many times of multiplying, I guess, overflow or underflow would be frequent.
I'm just curious whether deep learning library detect overflow or underflow operation and show any warning message, or just ignore it.