title: PyTorch: RuntimeError: Error(s) in loading state_dict for DETRModel

body:
I am trying to load the Facebook DETR with resnet50 pretrained model but I am getting the following error.
RuntimeError: Error(s) in loading state_dict for DETRModel:
    Missing key(s) in state_dict: "model.transformer.encoder.layers.0.self_attn.in_proj_weight", "model.transformer.encoder.layers.0.self_attn.in_proj_bias", "model.transformer.encoder.layers.0.self_attn.out_proj.weight", "model.transformer.encoder.layers.0.self_attn.out_proj.bias", "model.transformer.encoder.layers.0.linear1.weight", "model.transformer.encoder.layers.0.linear1.bias" ........

I am using a public dataset containing all the weights and models for DETR. I am using this dataset for the code.
This is the code I am using:
import sys
import torch.nn as nn
from pathlib import Path
sys.path.append('../input/detrmodels/facebookresearch_detr_master')

# copy pretrained weights to the folder PyTorch will search by default
Path('/root/.cache/torch/hub/').mkdir(exist_ok=True, parents=True)
Path('/root/.cache/torch/hub/checkpoints/').mkdir(exist_ok=True, parents=True)

detr_path = '/root/.cache/torch/hub/checkpoints/detr-r50-e632da11.pth'
resnet50_pretrained = '/root/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth'
detr_hub = '/root/.cache/torch/hub/facebookresearch_detr_master'

!cp ../input/detrmodels/detr-r50-e632da11.pth {detr_path}
!cp ../input/detrmodels/resnet50-19c8e357.pth {resnet50_pretrained}
!cp -R ../input/detrmodels/facebookresearch_detr_master {detr_hub}

DIR_INPUT = '../input/shopee-product-matching'
WEIGHTS_FILE = '../input/detrmodels/resnet50-19c8e357.pth'

class DETRModel(nn.Module):
    def __init__(self, num_classes, num_queries, model_name='detr_resnet50'):
        super(DETRModel, self).__init__()
        self.num_classes = num_classes
        self.num_queries = num_queries

        self.model = torch.hub.load('facebookresearch/detr', model_name, pretrained=True)

        self.in_features = self.model.class_embed.in_features

        self.model.class_embed = nn.Linear(in_features=self.in_features,
                                           out_features=self.num_classes)
        self.model.num_queries = self.num_queries

    def forward(self, images):
        return self.model(images)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
num_classes = 11011
num_queries = 100
model = DETRModel(num_classes=N_CLASSES, num_queries=num_queries, model_name='detr_resnet50')
model = model.to(device)
model.load_state_dict(torch.load(WEIGHTS_FILE))
model.eval()

How to fix this?


I am trying to calculate the precision and recall for two images. with one image as the output of the model (shape [3,786,1024]) and the other ground truth (shape [1, 786, 1024]). But I keep getting an error. I am new to python and I am able to understand what I am doing wrong. I have tried some things like getting the input and output to the same shape, but I am getting the error now:
raise ValueError("Classification metrics can't handle a mix of {0} "
ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass targets
If anybody can help me with this issue. i would be grateful.
My code is here:
def precision(outputs, labels):
op = outputs#.cpu()
la = labels#.cpu()
_, preds = torch.max(op, dim=1)
return torch.tensor(precision_score(la,preds, average='micro'))

def recall(outputs, labels):
op = outputs#.cpu()
la = labels#.cpu()
_, preds = torch.max(op, dim=1)
return torch.tensor(recall_score(la,preds, average='weighted', zero_division = 1))
def f1(outputs, labels):
op = outputs#.cpu()
la = labels#.cpu()
_, preds = torch.max(op, dim=1)
return torch.tensor(f1_score(la,preds, average='weighted'))

img50 = Image.open('0050_999.png')
numpydata_in = np.array(img50).astype('uint8')
#print("test shape",numpydata_in.shape)
numpydata_in= torch.from_numpy(numpydata_in)
print("final input shape", numpydata_in.shape)

imgT50 = Image.open('0050.jpg')
numpydata_out = np.array(imgT50).astype('uint8')
numpydata_out = transforms.ToTensor()(numpydata_out)
#print(numpydata_out.shape)
numpydata_out = torch.argmax(numpydata_out, dim=0)
#print(numpydata_out.shape)
numpydata_out = torch.squeeze(numpydata_out)
print("final shape target",numpydata_out.shape)

prec = precision(numpydata_in, numpydata_out)
print("precision is",prec)
rec = recall(numpydata_in, numpydata_out)
print("recall is",rec)
f1 = f1(numpydata_in, numpydata_out)
print("f1 score is",f1)

final input shape torch.Size([768, 1024])
final shape target torch.Size([768, 1024])
raise ValueError("Classification metrics can't handle a mix of {0} "
ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass targets


title: simulate nn.BatchNorm2d of pytorch with python code(batch normalization in pytorch)

body:
i want to simulate BatchNorm2d and write this code and compare the result with result of nn.BatchNorm2d function of pytorch when i print the output of my code to the output of nn.BatchNorm2d,each channels of  the output tensor denoted by hk1,hk2 is a coefficient of the output channels of the nn.BatchNorm2d function.i want to understand what are these coefficients?
import torch
import torch.nn as nn

m1 = nn.BatchNorm2d(2)
inp1 = torch.tensor([[[[1.,0.,3.,2.]],[[0.,2.,0.,4.]]],[[[0.,3.,0.,2.]],[[0.,1.,0.,1.]]],[[[0.,5.,0.,2.]],[[0.,1.,3.,1.]]]])

ot0 = torch.mean(inp1,(0,2,3))
jazr0 = torch.var(inp1,(0,2,3))

gt1 = inp1[:,0,:,:]
rd1=gt1-ot0[0]
hk1 = rd1/jazr0[0]

gt2 = inp1[:,1,:,:]
rd2=gt2-ot0[1]
hk2 = rd2/jazr0[1]

print('****batchnorm2d****:\n',m1(inp1))

btch0 = m1(inp1)[:,0,:,:]
btch1 = m1(inp1)[:,1,:,:]
print(hk1/btch0)
print(hk2/btch1)


title: Python dict + RuntimeError: Attempting to deserialize object on a CUDA device

body:
I was training ResNet-18 using PyTorch 1.8 and Python 3.8 where I saved the model metrics like val_loss, val_acc, acc, loss, etc. to a Python3 dict. After the training was done, I pickled the python dict for later analysis. Now, I am trying to load the pickled Python3 dict with the following code:
with open("ResNet18_cosine_annealing_scheduler_history.pkl", "rb") as file:
    d = pickle.load(file)

To which I get the error:

RuntimeError: Attempting to deserialize object on a CUDA device but
torch.cuda.is_available() is False. If you are running on a CPU-only
machine, please use torch.load with map_location=torch.device('cpu')
to map your storages to the CPU.

I then tried the code:
d = torch.load("ResNet18_cosine_annealing_scheduler_history.pkl", map_location='cpu')

But it gives the same error. Help?


title: I try to implement DQN with pytorch with OpenAI gym Environment. But I get a trouble that my episode_rewardis decrease. Why?

body:
Here are my code
The network input is the state(4d), output is the Q value(2d)
The Experience Replay I use the deque
Train
    for i in range(EPOCHS):
        episode_reward = 0
        done = False
        obs = env.reset()
        while not done:

            if random.random() < EPSILON:
                action = env.action_space.sample()
            else:
                with torch.no_grad():
                    q_values = np.array(policy_net(torch.tensor(np.array(obs)).float()))
                    action = np.argmax(q_values)

            next_obs, reward, done, info = env.step(action)

            episode_reward += reward

            queue.append((obs, action, reward, done, next_obs))

            if step_counter >= PRETRAINING_LENGTH:

                experience = queue.sample_mini_batch()

                #for the sampled experience minibatch 'experience'
                states, actions, rewards, dones, next_states = [i for i in experience]

                rewards_t = torch.FloatTensor(np.array(rewards).reshape(32, 1))
                actions_t = torch.LongTensor(np.array(actions).reshape(32, 1))
                dones_t = torch.FloatTensor(np.array(dones).reshape(32, 1))

                estimate = policy_net(torch.Tensor(states)).gather(1, actions_t)

                with torch.no_grad():
                    q_next = target_policy_net(torch.Tensor(np.array(next_states)))

                y_vector = rewards_t + DISCOUNT_FACTOR * q_next.max(1)[0].view(BATCH_SIZE, 1) * (1 - dones_t)

                optimizer.zero_grad()
                loss = functional.smooth_l1_loss(estimate, y_vector)
                loss.backward()
                optimizer.step()

            if step_counter % TARGET_NETWORK_UPDATE_FREQUENCY == 0:
                target_policy_net.load_state_dict(policy_net.state_dict()) # here we update the target policy network to match the policy network
            step_counter += 1

        obs = next_obs

        EPSILON = EPSILON * EPSILON_DECAY
        if EPSILON < MIN_EPSILON:
            EPSILON = MIN_EPSILON

        episode_reward_record.append(episode_reward)

        if i%100 ==0 and i>0:
            last_100_avg = sum(list(episode_reward_record))/100
            print("LAST 100 EPISODE AVERAGE REWARD: " + str(last_100_avg))
            print("EPSILON: " +  str(EPSILON))
            if last_100_avg > EARLY_STOPPING_THRESHOLD:
                break

I use a simple Network to approximate the Q function. And I use experience replay. The trouble I  have is that the episode_rewardis is decreasing as I training.


title: Why can I not update to the newest version of libtorch?

body:
I followed the tutorial here to make a c++ front-end application for pytorch:
https://pytorch.org/tutorials/advanced/cpp_frontend.html

I used GPU version. I could succesfully cmake and make.
Now I updated to the latest version of libtorch. I can still cmake, but on make, I get the following error:
[willem@int2 build]$ make
make[2]: *** No rule to make target `/sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/cuDNN/7.6.5.32-CUDA-10.1.243/include/lib64', needed by `polIter'.  Stop.
make[1]: *** [CMakeFiles/polIter.dir/all] Error 2
make: *** [all] Error 2

My cmake command is:
cmake -DCMAKE_PREFIX_PATH=~/libtorch1.8.1/share/cmake/Torch/
 -DCMAKE_MODULE_PATH=/home/willemvj/libtorch1.8.1/share/cmake/Torch/
-DCUDNN_LIBRARY_PATH=$EBROOTCUDNN/include/lib64
-DCUDNN_INCLUDE_DIR=$EBROOTCUDNN/include -DCUDNN_LIBRARY=$EBROOTCUDNN/lib64
/libcudnn.so.7 .. #

Any ideas on what is causing this?


title: Design paradigm for PyTorch - SSADM, OOD

body:
I have been developing a PyTorch model for a university project. In the thesis structure that the university has given me, they have mentioned a design paradigm. Structured systems analysis & design method and object-oriented design.
My question is that in Pytorch we have classes for the model, loss function, training, testing, and other classes like utils, Is that make the design paradigm of the project OOD? We have created objects for model and loss functions in the training class. just importing a util class to the training class makes an inheritance? The use of different classes for the model and the loss function and using them in training class can be called as abstraction?
Thank you!


title: Is there a python tensorflow/pytorch function like the x.toPixels tensorflowjs function?

body:
I need a replacement for the tf.browser.toPixels() tensorflowJS function. Trying to port some code to python and I'm wondering if there is a quick way around this.
In the browser this gets really simple and we just callback new frames and draw into a canvas. But in python development, say in matplotlib or tkinter, I guess I'm gonna need some tricks.
Is there a (not super big) solution for this?
Thanks


title: How to process batched data in PyTorch?

body:
I have a function which computes a new feature given several features of the data:
def my_func(f1, f2):
``
f1: torch.FloatTensor, shape = [1,2]
f2: torch.FloatTensor, shape = [1,2]
Returns
f3: torch.FloatTensor, shape = [1,2]
``
...

But my data is batched, so I have
f1 and f2's shape is [8,1,2]
I would like to apply my_func to get f3 with shape [8,1,2].
What is the idiomatic way to do this?


title: fill-mask usage from transformers pipeline

body:
I fine-tune a gpt2 language model and I am generation the text according to my model by using following lines of code:
generator = pipeline('text-generation', tokenizer='gpt2', model='data/out')
print(generator('Once upon a time', max_length=40)[0]['generated_text'])
Now I want to do the prediction of only next word with the probabilities. I know we can do it by using 'fill-mask' but I don't know how to do it. When I put 'fill-mask' inplace of 'text-generation', I am getting this error:
"Unrecognized configuration class <class 'transformers.models.gpt2.configuration_gpt2.GPT2Config'> for this kind of AutoModel: AutoModelForMaskedLM.
Model type should be one of BigBirdConfig, Wav2Vec2Config, ConvBertConfig, LayoutLMConfig, DistilBertConfig, AlbertConfig, BartConfig, MBartConfig, CamembertConfig, XLMRobertaConfig, LongformerConfig, RobertaConfig, SqueezeBertConfig, BertConfig, MobileBertConfig, FlaubertConfig, XLMConfig, ElectraConfig, ReformerConfig, FunnelConfig, MPNetConfig, TapasConfig, DebertaConfig, DebertaV2Config, IBertConfig.".
generator = pipeline('fill-mask', tokenizer='gpt2', model='data/out') // this line is giving me the above mentioned error.
Please let me know how can I fix this issue. Any kind of help would be greatly appreciated.
Thanks in advance.
The whole code for better understanding.
from transformers import (
GPT2Tokenizer,
DataCollatorForLanguageModeling,
TextDataset,
GPT2LMHeadModel,
TrainingArguments,
Trainer,
pipeline)
train_path = 'parsed_data.txt'
test_path = 'parsed_data.txt'
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
train_dataset = TextDataset(tokenizer=tokenizer, file_path=train_path, block_size=128)
test_dataset = TextDataset(tokenizer=tokenizer, file_path=test_path, block_size=128)
model = GPT2LMHeadModel.from_pretrained('gpt2')
training_args = TrainingArguments(output_dir = 'data/out', overwrite_output_dir = True, per_device_train_batch_size = 32, per_device_eval_batch_size = 32, learning_rate = 5e-5, num_train_epochs = 3,)
trainer = Trainer(model = model, args = training_args, data_collator=data_collator, train_dataset = train_dataset, eval_dataset = test_dataset)
trainer.train()
trainer.save_model()
generator = pipeline('fill-mask', tokenizer='gpt2', model='data/out')


title: is there any way for extracting audio features and save faster than this torch code?

body:
I have a pytorch code for extracting about 600 thousand audio features. For time-effficiency, I need to extract as fast as I can. I've tried several ways, and finally, I've come up with the following solution. Currently, I'm using a loop system by pytorch dataloader.
I wanted to know if there is a faster method to accomplish all of this. Is there some way that I can find the maximum during my loop of dataloader? Again, I'm not permitted to use inbuilt algorithms for searching or sorting. Any suggestions would help.
Thank you.
start_time = time.time()

dataset = AudioLoader(
    base_root=BASE_ROOT,
    date=DATE,
    data_type=DATA_TYPE,
    feat_type=FEAT_TYPE,
    params=PARAMS,
    samp_rate=SAMP_RATE
)

loader = DataLoader(
    dataset,
    batch_size=1,
    shuffle=False,
    num_workers=4
)

total_len = 0
mean = torch.zeros(FEAT_DIM)
sq_mean = torch.zeros(FEAT_DIM)
std = torch.zeros(FEAT_DIM)

for k, data in enumerate(loader):
    id = data['ID'][0]
    input_file = data['feat'][0]
    output_file = data['output'][0]

    if CAL_CMVN == True:
        total_len += input_file.shape[0]
        mean += torch.sum(input_file,0)
        sq_mean += torch.sum(torch.square(input_file), 0)

    save_features(input_file, WAV_PATH, output_file, OUTPUT_PATH, id)

    if k > 0 and k % 1000 == 0:
        print("-----{}s seconds-----".format(time.time() - start_time))

if CAL_CMVN == True:
    mean /= total_len
    sq_mean /= total_len
    std = torch.sqrt(sq_mean - mean*mean)

    torch.save(mean, os.path.join('./feat', 'mean.pt'))
    torch.save(std, os.path.join('./feat', 'std.pt'))

print("Total consumed time : {}s seconds".format(time.time() - start_time))

The function 'save_features' is for saving features in path. feature file is input file, and output file is a target of neural network for training input feature. Also, I can show you the detail of AudioLoader
class AudioLoader(Dataset):
    """Pytorch audio loader."""

    def __init__(
            self,
            base_root=BASE_ROOT,
            date="210223",
            data_type="train",
            feat_type="mfcc",
            params={
                "channel": -1,
                "dither": 0.0,
                "window_type": "hamming",
                "sample_frequency": 16000
            },
            samp_rate=16000
    ):
        """
        Args:
        """
        wavfile = os.path.join(base_root, date, "data", data_type) + ".scp"
        txtfile = os.path.join(base_root, date, "data", data_type) + ".txt"

        self.wavlist = open(wavfile, "r", encoding="UTF8").read().splitlines()
        self.txtlist = open(txtfile, "r", encoding="UTF8").read().splitlines()
        self.feat_type = feat_type
        self.samp_rate = samp_rate
        self.params = params

    def __len__(self):
        return len(self.wavlist)

    def __getitem__(self, idx):

        FEAT=torchaudio.compliance.kaldi.spectrogram

        if self.feat_type == "fbank":
            FEAT=torchaudio.compliance.kaldi.fbank
        elif self.feat_type == "mfcc":
            FEAT=torchaudio.compliance.kaldi.mfcc

        wav_id = self.wavlist[idx].split()[0]
        txt_id = self.txtlist[idx].split()[0]

        if wav_id != txt_id:
            raise Exception('Both id are unmatched. You shoud sort again.')

        wav_path = self.wavlist[idx].split()[-1]
        output_txt = self.txtlist[idx].split(' ', 1)[1]

        if os.path.isfile(wav_path):
            try:
                waveform, samp = torchaudio.load(wav_path)
                resamp_waveform = torchaudio.compliance.kaldi.resample_waveform(waveform, samp, self.samp_rate)
                feature = FEAT(resamp_waveform, **self.params)
            except:
                print('Extraction failed')
                return {'ID': wav_id}
        else:
            print('Extraction failed')
            return {'ID': wav_id}

        return {'ID': wav_id, 'feat': feature, 'output': output_txt}


title: Can make regression with PyTorch

body:
I'm trying to do a non linear regression with PyTorch with this code
import os; os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import matplotlib.pyplot as plt

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, 4)
        self.fc2 = nn.Linear(4, 4)
        self.fc3 = nn.Linear(4, 1)

    def forward(self, x):
        x = torch.sigmoid(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        x = self.fc3(x)
        return x

if __name__ == '__main__':

    net = Net()

    X = torch.tensor([[x] for x in np.arange(0, 10, 0.1)], dtype=torch.float32)
    Y = 10 * X**2

    learning_rate = 0.01
    EPOCHS = 100

    loss = nn.MSELoss()
    optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate)

    for epoch in range(EPOCHS):
        y_predicted = net(X)

        l = loss(Y, y_predicted)
        l.backward()
        optimizer.step()
        optimizer.zero_grad()

    with torch.no_grad():
        plt.plot(X.numpy(),Y.numpy())
        plt.plot(X.numpy(),net(X).numpy())
        plt.legend(['sample','prediction'])
        plt.show()

When I plot the prediction line I get a straight line rather than a curved one.

Please, can you help me with the regression?


title: How to solve RuntimeError: Cannot initialize CUDA without ATen_cuda library

body:
I'm on a MacBook Pro and have Intel Iris Plus Graphics 650 1536 MB. I am trying to run python3 code in an anaconda environment. All the code I am trying to run is not my own, it is from a GitHub repository. I have installed pytorch 0.4.1 and coda toolkit 9.0.
I am getting the following error:
RuntimeError: Cannot initialize CUDA without ATen_cuda library. PyTorch splits its backend into two shared libraries: a CPU library and a CUDA library; this error has occurred because you are trying to use some CUDA functionality, but the CUDA library has not been loaded by the dynamic linker for some reason.  The CUDA library MUST be loaded, EVEN IF you don't directly use any symbols from the CUDA library! One common culprit is a lack of -Wl,--no-as-needed in your link arguments; many dynamic linkers will delete dynamic library dependencies if you don't depend on any of their symbols.  You can check if this has occurred by using ldd on your binary to see if there is a dependency on *_cuda.so library.

This error occurs when I type the following in the terminal:
python3 run_lama.py --wikiname wikipedia2vec-base-cased --modelname bert-base-cased --data_dir ../data/LAMA/data --output_dir ../outputs/LAMA --infer_entity

I don't understand what this exactly means and how I can solve this. Googling the error message didn't help me. Does anyone have any suggestions?


This is my code and I use pytorch-ignite. The shape of sample's labels are (batch_size,) and the outputs of my netwroy as y_pred is (batch_size,10) and 10 is the number of my class. I use  criterion = F.cross_entropy as a loss function. after an epoch is finished, I have an error like:
Epoch [1/50]: 100%|██████████| 63/63 [00:55<00:00,  1.12it/s]
Current run is terminating due to exception: y and y_pred must have same shape of (batch_size, num_categories, ...) and num_categories > 1.
Engine run is terminating due to exception: y and y_pred must have same shape of (batch_size, num_categories, ...) and num_categories > 1.
Engine run is terminating due to exception: y and y_pred must have same shape of (batch_size, num_categories, ...) and num_categories > 1.
Traceback (most recent call last):
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-b8f3a45f8e35>", line 1, in <module>
    runfile('/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools/train_net.py', wdir='/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools')
  File "/home/omid/OMID/program/pycharm-professional-2020.2.4/pycharm-2020.2.4/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/omid/OMID/program/pycharm-professional-2020.2.4/pycharm-2020.2.4/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools/train_net.py", line 73, in <module>
    main()
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools/train_net.py", line 69, in main
    train(cfg)
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools/train_net.py", line 48, in train
    do_train(
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/engine/trainer.py", line 113, in do_train
    trainer.run(train_loader, max_epochs=epochs)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 752, in _internal_run
    self._fire_event(Events.EPOCH_COMPLETED)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/engine/trainer.py", line 73, in log_training_results
    evaluator.run(train_loader)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 745, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 850, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 834, in _run_once_on_dataset
    self._fire_event(Events.ITERATION_COMPLETED)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 424, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/metrics/metric.py", line 296, in iteration_completed
    self.update(output)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/metrics/metric.py", line 575, in wrapper
    func(self, *args, **kwargs)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/metrics/accuracy.py", line 149, in update
    self._check_shape(output)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/metrics/accuracy.py", line 47, in _check_shape
    raise ValueError(
ValueError: y and y_pred must have same shape of (batch_size, num_categories, ...) and num_categories > 1.

this is my debug part screenshot:

From what you can see the shape of y is (16,) and the shape of y_pred is (16,10) and both y_shape and y_pred_shape are 16

# encoding: utf-8

import logging

import torch
from ignite.contrib.handlers import ProgressBar
from ignite.engine import Events, create_supervised_trainer, create_supervised_evaluator
from ignite.handlers import ModelCheckpoint, Timer
from ignite.metrics import Accuracy, Loss, RunningAverage
from ignite.metrics.precision import Precision
from ignite.metrics.recall import Recall

from ignite.metrics.metrics_lambda import MetricsLambda

def do_train(
        cfg,
        model,
        train_loader,
        val_loader,
        optimizer,
        scheduler,
        loss_fn,
):
    log_period = cfg.SOLVER.LOG_PERIOD
    checkpoint_period = cfg.SOLVER.CHECKPOINT_PERIOD
    output_dir = cfg.OUTPUT_DIR
    device = cfg.MODEL.DEVICE
    device = torch.device("cuda")
    epochs = cfg.SOLVER.MAX_EPOCHS

    model = model.to(device)

    logger = logging.getLogger("template_model.train")
    logger.info("Start training")

    precision = Precision(average=True, is_multilabel=True)
    recall = Recall(average=True, is_multilabel=True)
    F1 = precision * recall * 2 / (precision + recall + 1e-20)
    F1 = MetricsLambda(lambda t: torch.mean(t).item(), F1)

    trainer = create_supervised_trainer(model, optimizer, loss_fn, device=device)
    evaluator = create_supervised_evaluator(model, metrics={'accuracy': Accuracy(is_multilabel=True),
                                                            'precision': precision,
                                                            'recall': recall,
                                                            'f1': F1,
                                                            'ce_loss': Loss(loss_fn)}, device=device)
    checkpointer = ModelCheckpoint(output_dir, 'music', n_saved=10, require_empty=False)
    timer = Timer(average=True)

    trainer.add_event_handler(Events.EPOCH_COMPLETED, checkpointer, {'model': model,
                                                                     'optimizer': optimizer})

    timer.attach(trainer, start=Events.EPOCH_STARTED, resume=Events.ITERATION_STARTED,
                 pause=Events.ITERATION_COMPLETED, step=Events.ITERATION_COMPLETED)

    RunningAverage(output_transform=lambda x: x).attach(trainer, 'avg_loss')

    pbar = ProgressBar(persist=True, bar_format="")
    pbar.attach(trainer)

    @trainer.on(Events.ITERATION_COMPLETED)
    def log_training_loss(engine):
        iter = (engine.state.iteration - 1) % len(train_loader) + 1

        if iter % log_period == 0:
            logger.info("Epoch[{}] Iteration[{}/{}] Loss: {:.2f}"
                        .format(engine.state.epoch, iter, len(train_loader), engine.state.metrics['avg_loss']))

    @trainer.on(Events.EPOCH_COMPLETED)
    def log_training_results(engine):
        evaluator.run(train_loader)

        metrics = evaluator.state.metrics

        avg_accuracy = metrics['accuracy']
        precision = metrics['precision']
        recall = metrics['recall']
        f1 = metrics['f1']

        avg_loss = metrics['ce_loss']

        logger.info(
            "Training Results - Epoch: {} Avg accuracy: {:.3f}, precision: {:.3f}, recall: {:.3f}, f1 score: {:.3f},  Avg Loss: {:.3f}"
                .format(engine.state.epoch, avg_accuracy, precision, recall, f1, avg_loss))

    if val_loader is not None:
        @trainer.on(Events.EPOCH_COMPLETED)
        def log_validation_results(engine):
            evaluator.run(val_loader)
            metrics = evaluator.state.metrics

            avg_accuracy = metrics['accuracy']
            precision = metrics['precision']
            recall = metrics['recall']
            f1 = metrics['f1']

            avg_loss = metrics['ce_loss']
            logger.info(
                "Validation Results - Epoch: {} Avg accuracy: {:.3f}, precision: {:.3f}, recall: {:.3f}, f1 score: {:.3f}, Avg Loss: {:.3f}"
                    .format(engine.state.epoch, avg_accuracy, precision, recall, f1, avg_loss)
            )

    # adding handlers using `trainer.on` decorator API
    @trainer.on(Events.EPOCH_COMPLETED)
    def print_times(engine):
        logger.info('Epoch {} done. Time per batch: {:.3f}[s] Speed: {:.1f}[samples/s]'
                    .format(engine.state.epoch, timer.value() * timer.step_count,
                            train_loader.batch_size / timer.value()))
        timer.reset()

    trainer.run(train_loader, max_epochs=epochs)


title: GAN is stuck during training right form the beggining

body:
I am trying to build a GAN using MNIST data-set, and when I start training it, I can see that both the generator and discriminator are stuck (.i.e. the loss values won't change and the generator outputs the same white noise meaningless images). I tried changing the loss functions for both the discriminator and generator, the learning rate, adding and removing layers but none of the attempts worked. Attached is my code:
def tensor_to_plt_im(im: torch.Tensor):
    return im.permute(1, 2, 0)

def d_loss(discriminator_generated_x, discriminator_true_x):
    return -0.5 * torch.mean(torch.log(discriminator_true_x + 1e-8)) \
           - 0.5 * torch.mean(torch.log(1 - discriminator_generated_x + 1e-8))

def gen_loss(discriminator_generated_x, discriminator_true_x):
    return -1 * d_loss(discriminator_generated_x, discriminator_true_x)

def gen_loss_non_saturating(discriminator_generated_x):
    return -0.5 * torch.mean(torch.log(discriminator_generated_x + 1e-8))

def d_loss_least_squares(discriminator_generated_x, discriminator_true_x):
    return 0.5 * torch.mean(torch.square(discriminator_true_x + 1e-8 - 1)) \
           + 0.5 * torch.mean(torch.square(discriminator_generated_x + 1e-8))

def gen_loss_least_squares(discriminator_generated_x):
    return 0.5 * torch.mean(torch.square(discriminator_generated_x + 1e-8 - 1))

class MNISTGen(nn.Module):
    def __init__(self, num_input_channels):
        super(MNISTGen, self).__init__()
        self.conv = nn.Sequential(
            # input is (num_input_channels*4) x 4 x 4
            nn.ConvTranspose2d(num_channels * 4, num_channels * 2, kernel_size=3, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(num_channels * 2),
            # nn.ReLU(True),
            # size. (num_channels*2) x 7 x 7
            nn.ConvTranspose2d(num_channels * 2, num_channels, kernel_size=4, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(num_channels),
            # nn.ReLU(True),
            # size. (num_channels) x 14 x 14
            nn.ConvTranspose2d(num_channels, num_input_channels, kernel_size=4, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(num_input_channels),
            nn.Sigmoid()  # the final output image size. (num_input_channels) x 28 x 28
        )
        self.linear = nn.Sequential(
            # size. (latent_vec_size)
            nn.Linear(latent_vec_size, 64),
            nn.BatchNorm1d(64),
            # nn.LeakyReLU(0.2, inplace=True),
            # size. (64)
            nn.Linear(64, 256),
            nn.BatchNorm1d(256),
            # size. 256
            nn.Linear(256, 512),
            nn.BatchNorm1d(512),
            # size. 512
            nn.Linear(512, (num_channels * 4) * 4 * 4),
            nn.BatchNorm1d((num_channels * 4) * 4 * 4),
        )

    def forward(self, x):
        x = self.linear(x)
        x = x.view(-1, num_channels * 4, 4, 4)
        return self.conv(x)

class MNISTDisc(nn.Module):
    def __init__(self, num_input_channels):
        super(MNISTDisc, self).__init__()
        self.conv = nn.Sequential(
            # input is (num_input_channels) x 28 x 28
            nn.Conv2d(num_input_channels, num_channels, kernel_size=4, stride=2, padding=1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # size. (num_channels) x 14 x 14
            nn.Conv2d(num_channels, num_channels * 2, kernel_size=4, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(num_channels * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # size. (num_channels*2) x 7 x 7
            nn.Conv2d(num_channels * 2, num_channels * 4, kernel_size=3, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(num_channels * 4),
            nn.LeakyReLU(0.2, inplace=True),  # size. (num_channels*4) x 4 x 4
        )
        self.linear = nn.Sequential(
            # size. (num_channels_d*4) x 4 x 4
            nn.Linear((num_channels * 4) * 4 * 4, 512),
            nn.BatchNorm1d(512),
            nn.LeakyReLU(0.2, inplace=True),
            # size. (512)
            nn.Linear(512, 256),
            nn.BatchNorm1d(256),
            # size. 256
            nn.Linear(256, 64),
            nn.BatchNorm1d(64),
            # size. 64
            nn.Linear(64, 1),
            nn.Sigmoid()  # final classification
        )

    def forward(self, x):
        x = self.conv(x)
        x = x.view(-1, (num_channels * 4) * 4 * 4)  # reshape input so that it fits fc layer
        return self.linear(x)

def train(g_net: nn.Module, d_net: nn.Module):
    G_losses, D_losses, iters, gen_lst = [], [], 0, []
    optimizerD = optim.Adam(d_net.parameters(), lr=lr, betas=(beta1, 0.999))
    optimizerG = optim.Adam(g_net.parameters(), lr=lr, betas=(beta1, 0.999))
    print("Starting Training Loop...")
    # For each epoch
    for epoch in range(num_epochs):
        # For each batch in the dataloader
        for i, data in enumerate(dataloader, 0):
            ############################
            # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
            ###########################
            # Train with all-real batch
            d_net.zero_grad()
            # Format batch
            real_cpu = data[0].to(device)
            b_size = real_cpu.size(0)
            # Forward pass real batch through D
            real_output = d_net(real_cpu).view(-1)
            # errD_real.backward()
            D_x = real_output.mean().item()

            # Train with all-fake batch
            # Generate batch of latent vectors
            noise = torch.randn(b_size, latent_vec_size, device=device)
            # Generate fake image batch with G
            fake = g_net(noise)
            # Classify all fake batch with D
            fake_output = d_net(fake.detach()).view(-1)
            # Calculate the gradients for this batch, accumulated (summed) with previous gradients
            loss = d_loss(fake_output, real_output)
            loss.backward()
            # errD_fake.backward()
            D_G_z1 = fake_output.mean().item()
            # Compute error of D as sum over the fake and the real batches
            errD = loss
            # Update D
            optimizerD.step()

            # perform a generator iteration every 'discriminator_iterations' steps of the discriminator
            if i % discriminator_iterations == 0:
                ############################
                # (2) Update G network: maximize log(D(G(z)))
                ###########################
                g_net.zero_grad()
                # Since we just updated D, perform another forward pass of all-fake batch through D
                fake_output = d_net(fake).view(-1)
                # Calculate G's loss based on this output
                # errG = criterion(fake_output, label)
                errG = gen_loss_non_saturating(fake_output)
                # Calculate gradients for G
                errG.backward()
                D_G_z2 = fake_output.mean().item()
                # Update G
                optimizerG.step()

                # Output training stats
                if i % 50 == 0:
                    print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                          % (epoch, num_epochs, i, len(dataloader),
                             errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))
                G_losses.append(errG.item())

            D_losses.append(errD.item())  # Save Losses for plotting later

            # Check how the generator is doing by saving G's output on fixed_noise
            if (iters % 500 == 0) or ((epoch == num_epochs - 1) and (i == len(dataloader) - 1)):
                with torch.no_grad():
                    fake = g_net(fixed_noise).detach().cpu()
                gen_lst.append(vutils.make_grid(fake))
                plt.imshow(tensor_to_plt_im(gen_lst[-1]))
                plt.show()
            iters += 1

if __name__ == '__main__':
    dataloader = generate_mnist_data_set()
    generator = MNISTGen(num_input_channels=1)
    discriminator = MNISTDisc(num_input_channels=1)

    generator.apply(weights_init), discriminator.apply(weights_init)

    # Create batch of latent vectors that we will use to visualize
    #  the progression of the generator
    fixed_noise = torch.randn(image_size, latent_vec_size, device=device)
    train(generator, discriminator)

I would greatly appreciate any suggestions, remarks and general directions because like I said, I am completely stuck.
Thanks in advance


title: pytorch nn loading checkpoint and steping scheduler

body:
Im trying to load a checkpoint in pytorch neural network and step down the learning rate with scheduler but it's not working and line #1 and line #2 print the same thing. sometimes it gives an error like this  1.1.0 changed this behavior in a BC-breaking way. If you use the learning rate scheduler (calling scheduler.step()) before the optimizer’s update (calling optimizer.step()), this will skip the first value of the learning rate schedule. and Im using this code at the end of my training loop in an if statement. if further informations you think is necessary please make a comment. cause I think it's enough right now.
print('<<<back to the best>>>')
loaded_checkpoint=torch.load("checkpoint.pth")
model = NeuralNet(input_size, hidden_size, output_size)
model.load_state_dict(checkpoint['model_state'])
model.to(device)
model.eval()

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
optimizer.load_state_dict(checkpoint['optim_state'])

print('lr',optimizer.state_dict()['param_groups'][0]['lr']) #1
losss=.9
scheduler.step(losss)
print('<<<back to the best>>>')
print('lr',optimizer.state_dict()['param_groups'][0]['lr']) #2


title: 100% CPU and computer crashes running TorchScript Module on CPU?

body:
I've gone through the process of quantizing and scripting a PyTorch module for the first time much in the same manner as this tutorial: https://leimao.github.io/blog/PyTorch-Static-Quantization/
The tutorial code runs fine, but when I translated this to my own encoder decoder module I saw some odd behaviour. So here are some relevant bits of code:
def measure_inference_latency(model, input_size, num_samples=100, num_warmups=10):
    model.eval()

    x = torch.rand(size=input_size)

    with torch.no_grad():
        for _ in range(num_warmups):
            _ = model(x)
    # torch.cuda.synchronize()

    with torch.no_grad():
        start_time = time.time()
        for _ in trange(num_samples):
            _ = model(x)
            # torch.cuda.synchronize()
        end_time = time.time()
    elapsed_time = end_time - start_time
    elapsed_time_ave = elapsed_time / num_samples

    return elapsed_time_ave

num_samples = 5
fp32_latency = measure_inference_latency(model_fp32, input_size = [1, 3] + CFG.image_dim, num_samples=num_samples)
q_latency = measure_inference_latency(model_q, input_size = [1, 3] + CFG.image_dim, num_samples=num_samples)
start = time.time()
qj_latency = measure_inference_latency(model_qj, input_size = [1, 3] + CFG.image_dim, num_samples=num_samples)
print(time.time() - start)

The output I get is:
FP32 inference latency: 269.17 ms
Quantized inference latency: 38.93 ms
Quantized + JIT inference latency: 24.36 ms

Seems reasonable, but here are the problems I'm noticing:

In wall-clock time, the qj_latency line runs for like 17 seconds. I narrowed this down to the warmup step in measure_inference_latency which is not included in the final time reported. When I added a progress bar to the warmup I noticed most of the time is in the first step. Is there a long warmup time needed for TorchScript modules?
On one occasion my whole computer just shut down (Ubuntu 20.04.2 LTS).

I know I haven't provided full code, but to start with I was wondering if there is an obvious answer I don't know about.


title: The loss value goes crazy after a while when training an OpenAI jukebox upsampler

body:
When training an upsampler with OpenAI Jukebox, at the end of the first epoch (after approx. 2 days) there were more "overflow in backward" messages and ultimately the loss value jumped by 20. What could be the problem? VQVAE training with the same data worked without any problems.

 98% 28837/29318 [51:38:38<51:38,  6.44s/it, bpd=5.07, g_l=5.07, gn=43.4, l=5.07, p_l=0]

Warning. Loss is 25.929014205932617

Overflow in backward. Loss 25.929014205932617, grad norm nan, lgscale -7.203999999995833, new lgscale -8.203999999995833

 98% 28838/29318 [51:38:45<51:46,  6.47s/it, bpd=5.07, g_l=5.07, gn=43.4, l=5.07, p_l=0]

Warning. Loss is 26.65635871887207

 98% 28838/29318 [51:38:52<51:46,  6.47s/it, bpd=26.7, g_l=26.7, gn=1.01e+8, l=26.7, p_l=0]
 98% 28839/29318 [51:38:52<53:08,  6.66s/it, bpd=26.7, g_l=26.7, gn=1.01e+8, l=26.7, p_l=0]```


title: Transforming the indices back through view in pytorch

body:
I have the following code, where I define a torch tensor of zeros, change one item to be equal to 1, pass through three reshape functions.
Then, after all the transformations, I obtain the index of 1. I am wondering if it is possible to somehow use the max_idx and the information about the permutations/.view to obtain the index of 1 in the initial B1 tensor (which should be equal to 1234).
A1 = np.zeros(10*18*40*28)
A1[1234] = 1
A1 = A1.reshape(10, 18, 40, 28)
B1 = torch.Tensor(A1)
print('B1: ', B1.shape, torch.nonzero(B1))
C1 = B1.permute(0, 2, 3, 1)
print('C1: ', C1.shape,  torch.nonzero(C1))
D1 = C1.contiguous().view(C1.shape[0], C1.shape[1], C1.shape[2], 3, 6)
print('D1: ', D1.shape, torch.nonzero(D1))
E1 = D1.contiguous().view(D1.shape[0], -1, 6)
print('E1: ', E1.shape, torch.nonzero(E1))

max_idx = torch.nonzero(E1)

I would love to hear any hints on how I can try to do this :)


title: Fail to build PyTorch at libtorch_cuda.so (using conda)

body:
I'm building PyTorch from source to get RTX3090 support (sm86) but it keeps error. The machine has Nvidia Driver 455+, Cuda 11.1 and CuDNN 8.1. I use conda and installed magma-cuda111-2.5.2. Anyone can help?
P/S: I'm thinking about the error caused by ld. the LD_LIBRARY_PATH is
/usr/local/cuda/lib64 in conda (I set it as a tutorial when install CUDA).
FAILED: bin/roi_align_op_gpu_test
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic caffe2/CMakeFiles/roi_align_op_gpu_test.dir/operators/roi_align_op_gpu_test.cc.o -o bin/roi_align_op_gpu_test  -Wl,-rpath,/usr/local/cuda/lib64:/home/hieupth/Downloads/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/hieupth/Downloads/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/hieupth/Downloads/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /usr/local/cuda/lib64/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `zgemm_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `sgebak_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `zgehd2_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `zgemv_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `zgetrf_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `dlascl_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `slartg_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `dger_'
....
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `zpotrf_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `sgeqrf_'
/usr/bin/ld: /home/hieupth/Downloads/pytorch/build/lib/libtorch_cuda.so: undefined reference to `icamax_'
collect2: error: ld returned 1 exit status


title: Predicting A Value In The Future With Pytorch

body:
I'm doing timeseries stock price analysis with an LSTM model, and after training, I want to predict future values.
The data set is 52 weekly Close candles, so 1 year in total. For the feature set I use a sliding window approach of the last 25 weeks/candles.
Data + Training Predictions Plot
How would I, for example get a close price prediction for 25 weeks in the future?
I already checked this answer, but its giving errors, and unknown variables.


title: LSTM Text Generation with Pytorch

body:
I am currently trying quote generation (character level) with LSTMs using Pytorch. I am currently facing some issues understanding exactly how the hidden state is implemented in Pytorch.
Some details:
I have a list of quotes from a character in a TV series. I am converting those to a sequence of integers with each character corresponding to a certain integer by using a dictionary char2idx. I also have the inverse of this idx2char where the mapping is reversed.
After that, I am using a sliding window, say of size window_size, and a step of size step to prepare the data.
As an example, let's say the sequence is [1, 2, 3, 4, 5, 0] where 0 stands for the EOS character. Then using window_size = 3 and step = 2, I get the sequence for x and y as:
x1 = [1, 2, 3], y1 = [2, 3, 4]
x2 = [3, 4, 5], y1 = [4, 5, 0]

x = [x1, x2], y = [y1, y2]

The next step is to train the model. I have attached the code I am using to train the model.
NOTE: I am not passing hidden states from one batch to the other as the ith sequence of the (j+1)th batch is probably not the next step to the ith sequence from the jth batch. (This is why I am using a sliding window to help the model remember). Is there a better way to do this?
My main question occurs during testing time. There are two methods by which I am testing.
Method 1:
I take the initial seed string, pass it into the model and get the next character as the prediction. Now, I add that to the starting string and pass this whole sequence into the model, without passing the hidden state. That is, I input the whole sequence to the model, with the LSTM having the initial hidden state as 0, get the output, append the output to the sequence and repeat till I encounter the EOS character.
Method 2:
I take the initial seed string, pass it into the model and get the next character as the prediction. Now, I just pass the character and the previous hidden state as the next input and continue doing so until an EOS character is encountered.
Question

According to my current understanding, the outputs of both methods should be the same because the same thing should be happening in both.
What's actually happening is that both methods are giving completely different results. Why is this happening?
The second one gets stuck in an infinite loop for most inputs (e.g. it gives "back to back to back to ....") and on some inputs, the first one also gets stuck. How to prevent and avoid this?
Is this related in some way to the training?

I have tried multiple different ways (using bidirectional LSTMs, using one hot encoding (instead of embedding), changing the batch sizes, not using a sliding window approach (using padding and feeding the whole quote at once).
I cannot figure out how to solve this issue. Any help would be greatly appreciated.
CODE
Code for the Model Class:
class RNN(nn.Module):
    def __init__(self, vocab_size, hidden_size, num_layers, dropout=0.15):
        super(RNN, self).__init__()
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.embedding = nn.Embedding(vocab_size, hidden_size)
        self.lstm = nn.LSTM(hidden_size, hidden_size, num_layers, dropout=dropout, batch_first=True)
        self.dense1 = nn.Linear(hidden_size, hidden_size*4)
        self.dense2 = nn.Linear(hidden_size*4, hidden_size*2)
        self.dense3 = nn.Linear(hidden_size*2, vocab_size)
        self.drop = nn.Dropout(dropout)

    def forward(self, X, h=None, c=None):
        if h is None:
            h, c = self.init_hidden(X.size(0))
        out = self.embedding(X)
        out, (h, c) = self.lstm(out, (h, c))
        out = self.drop(out)
        out = self.dense1(out.reshape(-1, self.hidden_size)) # Reshaping it into (batch_size*seq_len, hidden_size)
        out = self.dense2(out)
        out = self.dense3(out)
        return out, h, c

    def init_hidden(self, batch_size):
        num_l = self.num_layers
        hidden = torch.zeros(num_l, batch_size, self.hidden_size).to(DEVICE)
        cell = torch.zeros(num_l, batch_size, self.hidden_size).to(DEVICE)
        return hidden, cell

Code for training:
rnn = RNN(VOCAB_SIZE, HIDDEN_SIZE, NUM_LAYERS).to(DEVICE)
optimizer = torch.optim.Adam(rnn.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

rnn.train()
history = {}
best_loss = 100

for epoch in range(EPOCHS): #EPOCH LOOP
    counter = 0
    epoch_loss = 0

    for x, y in train_loader: #BATCH LOOP
        optimizer.zero_grad()
        counter += 1

        o, h, c = rnn(x)
        loss = criterion(o, y.reshape(-1))
        epoch_loss += loss.item()

        loss.backward()
        nn.utils.clip_grad_norm_(rnn.parameters(), 5) # Clipping Gradients
        optimizer.step()

        if counter%print_every == 0:
            print(f"[INFO] EPOCH: {epoch+1}, BATCH: {counter}, TRAINING LOSS: {loss.item()}")

    epoch_loss = epoch_loss/counter
    history["train_loss"] = history.get("train_loss", []) + [epoch_loss]
    print(f"\nEPOCH: {epoch+1} COMPLETED!\nTRAINING LOSS: {epoch_loss}\n")

Method 1 Code:
with torch.no_grad():
    w = None
    start_str = "Hey, "
    x1 = quote2seq(start_str)[:-1]

    while w != EOS_TOKEN:
        x1 = torch.tensor(x1, device=DEVICE).unsqueeze(0)
        o1, h1, c1 = rnn(x1)
        p1 = F.softmax(o1, dim=1).detach()
        q1 = np.argmax(p1.cpu(), axis=1)[-1].item()
        w = idx2char[q1]
        start_str += w
        x1 = x1.tolist()[0]+ [q1]

quote = start_str.replace("<EOS>", "")
quote

Method 2 Code:
with torch.no_grad():
    w = None
    start_str = "Are we back"
    x1 = quote2seq(start_str)[:-1]
    h1, c1 = rnn.init_hidden(1)

    while w != EOS_TOKEN:
        x1 = torch.tensor(x1, device=DEVICE).unsqueeze(0)
        h1, c1 = h1.data, c1.data
        o1, h1, c1 = rnn(x1, h1, c1)
        p1 = F.softmax(o1, dim=1).detach()
        q1 = np.argmax(p1.cpu(), axis=1)[-1].item()
        w = idx2char[q1]
        start_str += w
        x1 = [q1]

quote = start_str.replace("<EOS>", "")
quote


title: pytorch spyder variable explorer break

body: the line x=torch.from_numpy(x) which x is a numpy array at first, causes the spyder variable explorer to break. after that the no variable can be added to variable explorer eventhough they are usable. do you know any way to avoid this? I was using 5.0.2 and hadn't this problem but it appeared I don't know why, and upgraded spyder to 5.0.3 by windows installer and problem yet exists. or if you are using spyder please try and comment do you face the same problem? and I should say there is no error just a bug report pop up.


title: Vectorize for-loop - need to average slices of varying size

body:
I am trying to average subword embeddings to form a word-level representation. Each word has a corresponding start and end index, indicating which subwords make up that word.
sequence_output is a tensor of B * 384 * 768, where 384 is the max sequence length, and 768 is the number of features.
all_token_mapping is a tensor of B * 384 * 2, which contains a start and end index.
initial_reps is a tensor of num_nodes * 768, num_nodes is the sum of all the number of words (not subwords) in the different samples.
initial_reps = torch.empty((num_nodes, 768), dtype=torch.float32)
current_idx = 0
for i, feature_tokens_mapping in enumerate(all_token_mapping):
    for j, token_mapping in enumerate(feature_tokens_mapping):
        if token_mapping[0] == -1: # reached the end for this particular sequence
            break
        initial_reps[current_idx] = torch.mean(sequence_output[i][token_mapping[0]:token_mapping[-1] + 1], 0, keepdim=True)
        current_idx += 1

My current code will create an empty tensor of length num_nodes, and a for loop will calculate the values at each index, by checking token_mapping[0] and token_mapping[1] for the correct slice of sequence_output to average.
Is there a way to vectorize this code?
In addition, I have a list that holds the number of words for each sample. i.e. the sum of all the elements in the list == num_nodes


title: Rearranging PyTorch tensor in a windowed manner

body:
So I have a tensor
A = torch.tensor([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])

and I was hoping to rearrange it in sliding windows, that is,
f(A) = torch.tensor([[1,2,5,6],[3,4,7,8],[9,10,13,14],[11,12,15,16]])

by viewing every tensor in A as a 2-by-2 small window, and then arranging the "window"s in a 2-by-2 matrix. This operation is similar to ArrayFlatten of Mathematica, but I couldn' find a way to do it in PyTorch. Any help is welcome.


title: getitem for custom dataloader is not working

body:
I am new to deep learning and Pytorch. I have data set of 6000 images that have all four classes in a single folder. I have a .csv file that has an image name and there corresponding one hot encoded representation (Ground_Truth_Label). I want to bind the image name to its label, so I can feed it into the neural network. I found that a custom Data Loader can be used for this. I tried the following code snippet in PyCharm:

class DFUDataset(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.DFU = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.DFU)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
            img_name = os.path.join(self.root_dir, self.DFU.iloc[idx, 0])  # image names
            image = io.imread(img_name)
            img_label = os.path.join(self.root_dir, self.DFU.iloc[0, idx[1:]])  # image labels
            sample = {'image': image, 'img_label': img_label}
            return sample

DFU_dataset = DFUDataset(
    csv_file='C:/Users/aleems2/Desktop/dfu/DFUC2021_trainset_210427/DFUC2021_train/Labelled_data_ground_truth.csv',
    root_dir="C:/Users/aleems2/Desktop/dfu/DFUC2021_trainset_210427/DFUC2021_train/Labelled_test_images")

However, when I am trying to debug the code then, a blue circle comes right beside getitem (Blue circle), and the debugger does not go to this function. It just runs def init, and then comes out of class DFUDataset(Dataset). I do not know what to override getitem with.
I have spent hours on this, but could not figure it out.


title: CUDA error device-side assert error with Torchvision

body:
I've had a problem with Pytorch and CUDA which seems pretty common, where I get "RuntimeError: CUDA error: device-side assert triggered" when building a neural network with CUDA because the output of the model is not compatible with labels for calculating loss. I think this was caused because the output of the model had size [n, 1] instead of [n] where n is the batch size.
Using my own architecture, I could fix this by using .squeeze() in the forward pass function, to get the output to size [n]. Now I am using a pretrained ResNet model from torchvision, I haven't been able to achieve the same thing. I've tried defining my own module for the final layers but it hasn't worked:
class final_layers(nn.Module):
    def __init__(self):
        super(final_layers, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(512, 1),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.layers(x)
        return x.squeeze()

model = models.resnet18(pretrained=True)
model.fc = final_layers()
optimizer = Adam(model.parameters(), lr=0.0003)
criterion = BCELoss()
if torch.cuda.is_available():
    model = model.cuda()
    criterion = criterion.cuda()

This model works fine in Colab with CPU but with GPU I get the following error:
RuntimeError: CUDA error: device-side assert triggered

Any help would be greatly appreciated :)


title: Convert list of elements to mapped tensor

body:
I have an list of strings and a function that returns the proportion of equal text in each text.
example:
def proportion(text1, text2):
  ...

list_text = ['hey', 'hello', ..., 'bye'] (size n)

What I want is to calculate a tensor n by n with the result of the function N by N elements. Example:
result = [ 1/7 (hey, hey)   ...   3/4 (hey bye)]
         [ ...                                 ]
         [ 3/4 (hey bye)  ...     3/3 (bye bye)]

Is there any easy way?


title: Is reshape() same as contiguous().view()?

body: In PyTorch, for a tensor x is x.reshape(shape) always equivalent to x.contiguous().view(shape)?


title: CSV Class reader not getting actual image

body:
I have a CSV file reader which reads a csv file with one column and obtains data about the image such as its label. After much debugging I have found my all data about the image is read but I'm missing the actual get image part, so currently, cv2 shows the input as a black box.

Im not sure what to do to achieve this, and need some assistance

My CSV Reader Below

def __init__(self, csvPath, imageHeight, imageWidth, transform = None):
    """
    Arguments:
        A CSV File Path
        Path to Image Foldr
        Extension of Images
        PIL Transforms
    """
    self.dataFromCSV = pD.read_csv(csvPath)
    self.dataLabels = nP.asarray(self.dataFromCSV.iloc[:, 0])
    self.imageHeight = imageHeight
    self.imageWidth = imageWidth
    self.transform = transform

def __getitem__(self, index):
    singleImageLabel = self.dataLabels[index]
    imagePath = singleImageLabel.split(";")[0]
    print ('Path: ' + str(imagePath))
    originalImage = cv2.imread(imagePath)
    cv2.imshow('IMAGE', originalImage)

    # Create an Empty Numpy Array to Fill
    imageAsNumpy = nP.ones((32, 32), dtype = 'uint8')

    # Fill the Numpy Array with Data from Pandas DF
    for i in range(1):
        rowPosition = (i-1) // self.imageHeight
        columnPosition = (i-1) % self.imageWidth
        indexFirst = self.dataFromCSV.iloc[index][i].split(";")[3]
        indexLast = self.dataFromCSV.iloc[index][i].split(";")[6]
        imageAsNumpy[rowPosition][columnPosition] = indexFirst + indexLast
        print ('LABEL: ' + str(singleImageLabel))
        cv2.imshow('INPUT',imageAsNumpy)
        cv2.waitKey(0)
        cv2.destroyAllWindows()

    # Convert Image from Numpy Array to PIL Image, Mode 'L' is for Grayscale
    imageAsImage = Image.fromarray(imageAsNumpy)
    imageAsImage = imageAsImage.convert('1')

    # Transform Image to Tensor
    if self.transform is not None:
        imageAsTensor = self.transform(imageAsImage)

    # Transform Label to Tensor
    labelAsLabel = int(singleImageLabel.split(";")[7])
    labelAsTensor = torch.from_numpy(nP.array(labelAsLabel))
    # print ('Target: ' + str(labelAsTensor))

    # Return Image & the Label
    return (imageAsTensor, labelAsLabel)

def __len__(self):
    return len(self.dataFromCSV.index)


title: CNN Training Runs without error, but it does not display the results

body:
I am new to pytorch, and I am trying to train my model (CNN), using the following code:

The program runs fine, but it does not display this Epoch/Step/Loss/Accuracy part:

print(‘Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%’

as if (i+1) % 100 == 0: never turns to 0

Training part:

iter = 0

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(dataloaders['train']):
        images = Variable(images)
        labels = Variable(labels)

        # Clear the gradients
        optimizer.zero_grad()

        # Forward propagation
        outputs = model(images)

        # Calculating loss with softmax to obtain cross entropy loss
        loss = criterion(outputs, labels)

        # Backward prop
        loss.backward()

        # Updating gradients
        optimizer.step()

        iter += 1

        # Total number of labels
        total = labels.size(0)

        # Obtaining predictions from max value
        _, predicted = torch.max(outputs.data, 1)

        # Calculate the number of correct answers
        correct = (predicted == labels).sum().item()

        # Print loss and accuracy

        if (i+1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
                  .format(epoch + 1, num_epochs, i + 1, len(dataloaders['train']), loss.item(),
                          (correct / total) * 100))

Full Code:

https://pastebin.com/dshNmhRL


title: Computation of gradients

body:
I want to compute the gradient in the following scenario:
y = w_0x+w_1 and z = w_2x + (dy/dx)^2

    w = torch.tensor([2.,1.,3.], requires_grad=True)
    x = torch.tensor([0.5], requires_grad=True)
    y = w[0]*x + w[1]
    y.backward()
    l = x.grad
    l.requires_grad=True
    w.grad.zero_()
    z = w[2]*x + l**2
    z.backward()

I expect [4, 0, 0.5] instead I get [0, 0, 0.5]. I know in this case I can replace l by w_0 but, l can be a complex function of x in which case it is important that I compute the gradients numerically instead of changing the expression for z. Please let me know what changes I need to do get the correct gradient w.r.t w


title: maintaining dataloader thread alives with pytorch

body: I am using 3 dataloaders in a loop with the same numbers of workers but it's appears that the thread are killed and reinstantiated. is there a way to keep them alive for the next iteration?


title: CNN external window

body: I'm developing a CNN application but I need to capture an external window for live pixel processing, I've looked at pywin32 but that doesn't seem to work for me. I just need to be pointed in a direction to accomplish this.


title: PyTorch reshepe 4d to 2d, with "with saving 1 dim"

body:
Have a tensor X:

X.shape = [2, 3, 4, 5]

This tensor represents pictures.
X[0] - number of pic.
X[1] -  Chanels
X[2],X[3] -  Height and Width.

I want to convert this tensor to shape [40,2] where second dim is 3 channels, and first all pixels in input data, which correspond channels in right order.
try to use

reshape(-1,3)

but chanels are in incorrect order.
How i can convert it?
It is also interesting how i can to convert back, from [40,2] to [2,3,4,5]

SOLUTION

Find solution fo my case:

x.permute(1,0,2,3).reshape(3,-1).T

from this X:

to this X:


title: How to add a Activation Layer with a specified value in DeepLearning4J?

body: For example, the default value of alpha in Activation-ELU is 1. How to set it to 1.5(say). In other framework like PyTorch we can do this using torch.nn.ELU(1.5). I cannot find any documentation in Deeplearning4J.


title: Is there an elegant way to set different learning rate for different epoch?

body:
For example, the learning rate is 0.08 for the epoch in [1, 20]. And it turns to 0.02 for the epoch in [21, 50], and then it turns to 0.01 for the epoch in [51, 100].
Thank you very much for any help!


title: how to get the image_path of any sample from dataset object using skorch?

body:
Using Pytorch Datasets I can create a dataset object that yields (X[i], y[i], image_path[i]). In such way, I can map each item of the dataset with the path where it has been downloaded.
I tried to do that using skorch but skorch dataset object restricts the output to (X[i], y[i]). How can I recover the image_path for any item of the skorch dataset?
I found a solution, but I don't think it should be the proper way for do that. Such solution consists in create a dataframe ordering the rows exactly in the same way pytorch dataset do in the backend...in such way I can map to each other using indexes (0, 1, 2...) (if such order is not changed in other backend steps)


title: Is there a way to get a tree of objects currently in GPU memory in Pytorch?

body: So I try to clear up my GPU memory, yet something is stuck in there. How to  get a tree of objects currently in GPU memory with Pytorch?


title: AttributeError in extended class of torch.autograd.Function

body:
I am writing an extended class of torch.autograd.Function shown below:
class Binarizer(torch.autograd.Function):
    """Binarizes {0, 1} a real valued tensor."""

    def __init__(self, threshold=DEFAULT_THRESHOLD):
        super(Binarizer, self).__init__()
        self.threshold = threshold
        print(self.threshold) # This prints

    @staticmethod
    def forward(self, inputs):
        outputs = inputs.clone()
        print(self.threshold) # This does not!!
        outputs[inputs.le(self.threshold)] = 0
        outputs[inputs.gt(self.threshold)] = 1
        return outputs

    @staticmethod
    def backward(self, gradOutput):
        return gradOutput

I am getting an error message: AttributeError: 'BinarizerBackward' object has no attribute 'threshold' when I call forward pass of the class.
threshold_fn = Binarizer(threshold=threshold) # This executes the first print
mask_thresholded = threshold_fn.apply(self.mask_real) # This fails

Please help.


title: Can I train a model using the notebook instance of AWS sagemaker?

body: I want to train a PyTorch based model using AWS SageMaker but I don't know how to use the ML Training services. So I was wondering if it is possible to simply train the model using the notebook instance and uploading the model to the S3 bucket? Also, how would I ensure That I do not end up paying more than I need to?


title: regarding pytorch gpu installation

body:
On my system the output of cat /usr/local/cuda/version.txt is CUDA Version 9.0.176,nvidia-smi returns driver number as 384.13.
I am trying to install pytorch with GPU support using the following command
conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
However when I execute
import torch
torch.cuda.is_available()

I get the following message
assertion error
torch with GPU is not enabled
Any ideas why this is happening,and how do I rectify this error.


title: use same gradient to maximize one part of the model and minimize another part of the same model

body:
I want to calculate the gradient and use the same gradient to minimize one part and maximize another part of the same network (kind of adversarial case). For me, Ideal case would be, if there are two optimizers responsible for two part of the network/model and one of the optimizers has a negative learning rate. But it seems that PyTorch does not allow negative learning rate.
In this case what I am doing is:
loss.backward()
optimzer_for_one_part of the model.step()

and then
(-loss).backward()

Problem is, This time the again calculated gradient will not be the same(values are different but flopped of course) because some weights of the same network (same computation graph) have already been changed. But, Ideally, I want to use the flipped version of the previous gradient.
How can I achieve this?


title: Why do `Distribution` classes in PyroPPL have a `log_prob()` method but not a `prob()` method?

body:
In PyroPPL, there exists an abstract Distribution class  that has a sample() method and a log_prob() method. The log_prob() method allows you to evaluate the log probability density for the distribution for a given scalar or array of values. I'm surprised that there is no prob() method though. This means that whenever you wish to evaluate the probability density you need to daisy chain a log_prob() method and a exp() method. What is the reason for not having a prob() method? Is it right to assume that evaluating probability densities for a given distribution would be a common operation?
I am fairly new to PPL's so is this a common choice in other PPL's?


title: Simple way to read three files (input IFM, filters, and Conv2D output) and verify the output using the Tensorflow/Pytorch Conv2D reference?

body:
There are 3 files.

Input image (channel, height, width) shape listed in row-major format (width dimension varying fasted) with each value listed in a new line. So an image with 32 channels, 5 height, 10 width will have 32x5x10 lines.
Filter/weights specified in 2nd file in the same form.
output specified in the 3rd file in similar form as above.

Is there a simple pytorch/TF script one could write using Conv2D function of these frameworks to verify the output?

Or is there an easy way to use the reference Conv2D function in the C/C++ code?


title: pytroch multi datasets can not be released?

body:
I'd like to implement Word2vec on pythorch . Because of the data is too large to load into Memory at once, I split the data into several data files. At each time I using the data, I will load the data and create a dataset and dataloader w.r.t the file (Here I use dataloader to shuffle and get minibatchs). However, after finish some iterations (an data file is used completely), I failed to release the dataset w.r.t the data file, Which will hold the memory all the time , How can I release the dataset. The code is shown as follows:
for file_id in range(0,file_nums):
        dataset = MyDataset(file_paths[file_id])
        dataloader = DataLoader(dataset, batch_size=args.mb, shuffle=True)
        for  minibatch in dataloader:
            iword, owords, opower = minibatch
            loss = sgns(iword, owords,opower)
            optim.zero_grad()
            loss.backward()
            optim.step()
            loss.detach()
        del dataset
        gc.collect() # dataset can not be released


title: image analysis - locate pixel groups - desperately need efficient loop

body:
i'm implementing my own face detector on pytorch.
currently stuck having a matrix of probabilities for a face being in some location.
i'm trying to write an efficient loop that will count the number of consecutive non zero pixels and tell me  how many pixels are there in each consecutive group.

so in this example i would get let's say a dictionary blobs = {2:[(8,5),(9,6)],1:[(7,0)],21:[(1,9),(2,7),(2,8),(2,9),(2,10),(3,7)...,]}
len(dict) = #number of separate pixels groups
dict.keys() = #how many pixels in each group
dict.values() = #location of each pixel in said group

i started making a solution but i have a feeling there is a more elegant one that might be found with the help of this great forum (:


title: How to apply multiheaded attention to video data

body:
I understand this question is very general, but I was wondering if anyone has good resources for how to apply pytorch multiheaded attention (https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html) to video data of input dimensions Batch x Num Frames x Channels x Height x Width.
Specifically, I am not sure what the embedding dimension should be in using pytorch multihead and how I should create the query mask.
Thank you!


title: SSH Terminal Environment

body:
[I 2020-12-09 09:15:49,277] Trial 2 finished with value: 0.678097665309906 and parameters: {'num_filters1': 16, 'num_filters2': 16, 'kernel_size': 6, 'dropout_rate': 0.5876482728988799, 'optimizer': 'RMSprop', 'lr': 9.014890468942645e-05, 'Weight_decay': 2.6662100371168874e-09}. Best is trial 1 with value: 0.6492913365364075.
/home/shar/anaconda3/lib/python3.7/site-packages/optuna/structs.py:27: FutureWarning: `structs` is deprecated. Classes have moved to the following modules. `structs.StudyDirection`->`study.StudyDirection`, `structs.StudySummary`->`study.StudySummary`, `structs.FrozenTrial`->`trial.FrozenTrial`, `structs.TrialState`->`trial.TrialState`, `structs.TrialPruned`->`exceptions.TrialPruned`.
  warnings.warn(_message, FutureWarning)
Traceback (most recent call last):
  File "OptunaTest", line 573, in <module>
    n_instances, n_features, scores = run_analysis()
  File "OptunaTest", line 354, in run_analysis
    pruned_trials = [t for t in study.trials if t.state == optuna.structs.TrialState.PRUNED]
  File "OptunaTest", line 354, in <listcomp>
    pruned_trials = [t for t in study.trials if t.state == optuna.structs.TrialState.PRUNED]
  File "/home/shar/anaconda3/lib/python3.7/site-packages/optuna/__init__.py", line 54, in __getattr__
    return getattr(self._load(), item)
  File "/home/shar/anaconda3/lib/python3.7/site-packages/optuna/__init__.py", line 49, in _load
    module = importlib.import_module(self._name)
  File "/home/shar/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/shar/anaconda3/lib/python3.7/site-packages/optuna/structs.py", line 41, in <module>
    "This class was moved to :mod:`~optuna.trial`. Please use "
  File "/home/shar/anaconda3/lib/python3.7/site-packages/optuna/_deprecated.py", line 77, in deprecated
    removed_version = _get_removed_version_from_deprecated_version(deprecated_version)
  File "/home/shar/anaconda3/lib/python3.7/site-packages/optuna/_deprecated.py", line 39, in _get_removed_version_from_deprecated_version
    return "{}.0.0".format(parsed_deprecated_version.major + 2)
AttributeError: 'Version' object has no attribute 'major'

I somehow screwed up the SSH terminal pytorch environment and it's not able to find the attribute major for version object.
The code is able to run but towards the end the error above appeared. I reinstalled the torch package and the issue is still not resolved.
Anyone able to advise how to fix it?


title: How two calculate the except set between two tensors in pytorch?

body:
Given two tensors:
A = torch.tensor([1,2,3])
B = torch.tensor([3,5])

Then, I want to obtain the tensor C = tensor([1,2]), in which contains all elements of A except the elements of B. In numpy, I can use np.setdiff1d(A,B) to reach it, but how can I do with pytorch api?


title: Calling multiple functions in each pytorch multiprocess

body:
I am using the pytorch multiprocessing package.
Currently, my multiprocessing section lnum_processes = mp.cpu_count()
model.share_memory()

processes = []

for epoch in range(1, epochs + 1,num_processes):

    for rank in range(num_processes):
        p = mp.Process(target = train, args = (model,train_loader,optimizer,epoch+rank))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
    losses.append(test(model, val_loader))

torch.save(model.state_dict(), path)

I also have elsewhere a train function and a test function. Is there a way to have each process
also call both a train and a test function? Something like
num_processes = mp.cpu_count()

model.share_memory()

processes = []

for epoch in range(1, epochs + 1,num_processes):

    for rank in range(num_processes):
        p = mp.Process(target = (train,test), args = ((model,train_loader,optimizer,epoch+rank),(model, val_loader)))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
    losses.append(test(model, val_loader))

torch.save(model.state_dict(), path)

Thanks


title: 2-D Vectorized Function with pytorch

body:
I have a function that I need to compute at every entry of a 2-D torch tensor, and it is depends on the value of the index of both axes. Right now, I have only been able to implement this as a nested for loop, iterating over both axes. This is quite slow (and needs to be executed >10^5 times) and I want to speed it up for better scaling.
vs = 200
nt = 12
b = torch.ones(vs)/vs
n_kw = torch.rand((nt, vs))
n_k = torch.rand((nt,))

def estimate_p(nt, vs, n_kw, n_k):
    p = torch.zeros((nt, vs))

    for i in range(0, nt):
        for j in range(0, vs):
            p[i,j] = (n_kw[i, j] + b[j])/(n_k[i] + torch.sum(b))
    return p

Is there a way to vectorize this/map based in the i,j index?


title: Is partial 3D convolution or transpose+2D convolution faster?

body:
I have some data of shape B*C*T*H*W. I want to apply 2d convolutions on the H*W dimension.
There are two options (that I see):

Apply partial 3D convolution with shape (1, 3, 3). 3D convolution accepts data with shape B*C*T*H*W which is exactly what I have. This is however a pseudo 3d conv that might be under-optimized (despite its heavy use in P3D networks).

Transpose the data, apply 2D convolutions, and transpose the result back. This requires the overhead of data reshaping, but it makes use of the heavily optimized 2D convolutions.

data = raw_data.transpose(1,2).reshape(b*t, c, h, w).detach()
out = conv2d(data)
out = out.view(b, t, c, h, w).transpose(1, 2).contiguous()

Which one is faster?
(Note: I have a self-answer below. This aims to be a quick note for people who are googling, aka me 20 minutes ago)


title: Loss exploding by backward pass

body:
def weighted_mse_loss(input,target,weights):
    out = (input-target)**2
    #print('Simple-Ou-Shape::',out.shape,out)
    out = out * weights.expand_as(out)
    out=out.squeeze(0)
    loss = out.sum(0)
    loss=loss.sum() # or sum over whatever dimensions
    return loss

gen.train()
mse_loss=torch.nn.MSELoss()
for i,(hr,lr) in enumerate(train_dataloader):
  hr=hr.to(device)
  lr=lr.to(device)

  gen_optimizer.step()
  generated=gen(lr)

  #Target-Text-Regions

  with torch.no_grad():
    y,feature=craft_model(hr)
    y_refiner = refine_net(y, feature)
    score_link = y_refiner[0,:,:,0].cpu().data.numpy()
    score_link=np.expand_dims(score_link,0)
  score_link=torch.from_numpy(score_link)
  score_link=F.interpolate(score_link.unsqueeze(0),size=(hr[0][0].shape))
  region_map_loss=weighted_mse_loss(generated,hr,score_link)

  simple_mse_loss=mse_loss(generated,hr)
  print('Simple-MSE-Loss::',simple_mse_loss)
  print('RegionMapAware Loss::',region_map_loss)
  simple_mse_loss.backward(retain_graph=True)
  gen_optimizer.step()
  if i==10:
    break

I want to focus the loss on textual parts, so I am using a text detector model(CRAFT), and trying to mask the loss to the regions predicted by this model.
But my loss is not decreasing rather increasing exponentially
This is the progress-output
Simple-MSE-Loss:: tensor(1.7920, grad_fn=<MseLossBackward>)
RegionMapAware Loss:: tensor(1162.5378, grad_fn=<SumBackward0>)
Simple-MSE-Loss:: tensor(30369398., grad_fn=<MseLossBackward>)
RegionMapAware Loss:: tensor(1.6868e+10, grad_fn=<SumBackward0>)
Simple-MSE-Loss:: tensor(10484.7695, grad_fn=<MseLossBackward>)
RegionMapAware Loss:: tensor(6352537., grad_fn=<SumBackward0>)
Simple-MSE-Loss:: tensor(1.6715e+30, grad_fn=<MseLossBackward>)
RegionMapAware Loss:: tensor(1.0599e+33, grad_fn=<SumBackward0>)
--------------------------------------------------------------------

Thanks in advance for any help.


title: How to test on custom question for trained model in TensorFlow BERT

body:
I am fairly new to TensorFlow
here is my notebook
https://colab.research.google.com/drive/170XIXgEudD5t9ah2TFIYoewhL_0aUHmQ?usp=sharing
I have trained it to detect emphatic and non emphatic sentences
I want to test the model on a custom sentence to the trained model and want to get the prediction out of it.
like: Predict("Question-text")


title: When does one have to call share_memory_() in Pytorch when using distributed training?

body:
I want to train things in parallel/distributed in 1 machine using multiple CPUS or GPUs. Thus my question is when does one ever run:
        # I DONT THINK THIS IS NEEDED, since you don't need to share data...each process loads in own stuff and uses it
        x, y = x.share_memory_(), y.share_memory_()

as my comment says, my belief is that .share_memory_() is never actually needed in pytorch because:

when using gpu you have to move the data to the correct GPU anyway so you would do x.to(rank).
when using cpus, each process would have a copy of their own data (sending data is expensive in distributed training as far as I understand), so there is no need to have it "shared" with .shared_memory_() since they are already reading the same data from disk (or something similar to that I assume).

Are these assumptions correct? Is it true that for data we never need .memory_share_()? Btw the docs say:

Moves the underlying storage to shared memory.

This is a no-op if the underlying storage is already in shared memory and for CUDA tensors. Tensors in shared memory cannot be resized.

Note however that if one is using CPUs only for the model I think we do need:
    else:
    # if we want multiple cpu just make sure the model is shared properly accross the cpus with shared_memory()
    # note that op is a no op if it's already in shared_memory
        model = model.share_memory()
        ddp_model = DDP(model)  # I think removing the devices ids should be fine...?

related:

How to parallelize a training loop ever samples of a batch when CPU is only available in pytorch?


title: HPC cluster: How to figure out if the problem is in the cluster or in my code

body:
Our University just got a new cluster. I am trying to train multi-node multi-gpu neural network on it. Now, I am getting weird gpu_mut and gpu_mused utilisation. They are not consistent with each other.
See below -

Now, to figure out whether the problem is in my code or on hardware, I tried to run horovod's pytorch benchmark example  but I am getting

RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 1; 15.78 GiB total capacity; 2.68 GiB already allocated; 53.75 MiB free; 77.13 MiB cached; 0 bytes inactive)

My University IT support team is telling that the error is not from their side on seeing the inconsistent behaviour of gpu utilisation. Can anyone please point me how to approach so that whether I reach to a conclusion that either the bug is in my code or from the hardware arrangement of cluster.
The environment is
**Environment:**
Framework: PyTorch
Framework version: 1.3.1
Horovod version: 0.19.0
MPI version: 10.3.0.01rc04
CUDA version: 10.1.105
NCCL version: 2.5.6
Python version: 3.7.9
OS and version: Red Hat Enterprise Linux Server" VERSION="7.6 (Maipo)"
GCC version: 4.8.5 20150623 (Red Hat 4.8.5-36)

Also, it is IBM Power 9 cluster.
Full log of error -

[WARN DDL-2-17] Not performing connection tests. Cannot find 'mpitool' executable. This could be because you are using a version of mpi that does not ship with mpitool.

Please see /tmp/DDLRUN/DDLRUN.ETfOvOUTgv5U/ddlrun.log for detailed log.
+ /home/rxs1576/.conda/envs/opt/bin/mpirun -x LD_LIBRARY_PATH -x LSB_JOBID -x LSB_MCPU_HOSTS -x PATH -disable_gdr -gpu -mca plm_rsh_num_concurrent 4 --rankfile /tmp/DDLRUN/DDLRUN.ETfOvOUTgv5U/RANKFILE -n 8 -x DDL_HOST_PORT=2200 -x "DDL_HOST_LIST=t035:0,1;t055:2,3;t056:4,5;t082:6,7" -x "DDL_OPTIONS=-mode p:2x4x1x1 " bash -c 'source /share/apps/ibm_wml_ce/1.6.2/anaconda3/etc/profile.d/conda.sh && conda activate /home/rxs1576/.conda/envs/opt > /dev/null 2>&1 && /home/rxs1576/latest_scripts/launch.sh python /home/rxs1576/latest_scripts/benchmark_pytorch.py'
Traceback (most recent call last):
  File "/home/rxs1576/latest_scripts/benchmark_pytorch.py", line 102, in <module>
    timeit.timeit(benchmark_step, number=args.num_warmup_batches)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/timeit.py", line 233, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/timeit.py", line 177, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 6, in inner
  File "/home/rxs1576/latest_scripts/benchmark_pytorch.py", line 83, in benchmark_step
    output = model(data)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torchvision/models/resnet.py", line 204, in forward
    x = self.layer4(x)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torchvision/models/resnet.py", line 110, in forward
    identity = self.downsample(x)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 1; 15.78 GiB total capacity; 2.68 GiB already allocated; 53.75 MiB free; 77.13 MiB cached; 0 bytes inactive)
Traceback (most recent call last):
  File "/home/rxs1576/latest_scripts/benchmark_pytorch.py", line 102, in <module>
    timeit.timeit(benchmark_step, number=args.num_warmup_batches)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/timeit.py", line 233, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/timeit.py", line 177, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 6, in inner
  File "/home/rxs1576/latest_scripts/benchmark_pytorch.py", line 83, in benchmark_step
    output = model(data)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torchvision/models/resnet.py", line 201, in forward
    x = self.layer1(x)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torchvision/models/resnet.py", line 106, in forward
    out = self.conv3(out)
  File "/home/rxs1576/.conda/envs/opt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
"TEST_350029.err" 197L, 16215C                                                                                1,1           Top

1:


title: Vnet Deep learning 3D volumetric Image Segmentation. How to handle large and different size images?

body:
I have a V-Net to segment an organ from bunch of CT images. However, the training data varies in depth:
some examples [Batch, Channels, D, H, W]:
[32, 1, 34, 512, 512]
[32, 1, 125, 512, 512]
[32, 1, 80, 512, 512]

The training dataloader requires the same dimensions to be passed into the network. I have tried to implemented a random spatial crop [128, 128, 64] for the dataloader transform, partly due to memory issues, but the results are all funny (I think most of the crop aren't within the desired organ of interest, so all outputs are just zeros). Any suggestions for a work around?
I was thinking maybe preprocessing the data first and crop depth-wise by looking at the labels to find the desired depth to crop at.


title: Does dropped nodes randomly change when using dropout in pytorch independent of random seed?

body:
If I set the random seed using torch.manual_seed(1111), does the dropped out (eliminated) nodes will always be the same when using torch.nn.Dropout in every epoch?
Let's say I have 10 nodes in a layer and I use Dropout of 0.2 on that layer, I know that during model training about 2 of these nodes will be randomly eliminated. Now, if I set the random seed manually, does that mean that the same 2 nodes will be eliminated in every epoch? In other words, does torch.manual_seed effect torch.nn.Dropout?


title: Spotlight recommender. How can I train a model when adding a new user or item?

body:
I am work with Spotlight (a library for recommender systems, based around PyTorch), and after training the model, I add a new user or item to the Interactions dataset and then try to train the model, but I get the following error: ValueError: Maximum user id greater than number of users in model.
Is there any way to add users and item to the dataset and train the model online after that?


title: CNN-RNN - how to manage gradients?

body:
I'm trying to create a system that will perform action recognition. I'm trying to use a combination of CNN and RNN. For CNN I am using ShuffleNet v2. I replaced the fc of the model with my custom nn.Sequential:
    rcnn_layer = nn.Sequential(
        nn.Linear(1024, 128),
        View((1, 128)),
        InputModifier(10),
        nn.RNN(input_size=128, hidden_size=64),
        GetLastHidden(),
        nn.Linear(64, 3),
        nn.Softmax(dim=-1)
    )

The custom classes are:
View - reshapes the output of the first linear layer so that it fits the RNN
InputModifier - retains previous outputs of the model
GetLastHidden - returns last layer of the RNN
In the future I'd like to use the model on a phone (somehow), so I'm trying to make it as efficient as I can. Therefore, in order to make it run faster, in my InputModifier class I'm retaining the previous outputs of the CNN. Here's the code:
class InputModifier(nn.Module):
    prev = []

    def __init__(self, max_seq_len):
        assert max_seq_len != 0, '`max_seq_len` cannot be 0.'
        super(InputModifier, self).__init__()
        self.max_seq_len = max_seq_len

    def forward(self, x):
        self.prev.append(x)
        if self.max_seq_len > 0:
            self.prev = self.prev[-self.max_seq_len:]

        inp = torch.cat(self.prev)
        return inp

The issue is: I don't know how to get it to work. When I run loss.backward(), I get RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. error.
If i set loss.backward(retain_graph=True), I get RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1024, 128]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). .
PS: The training code:
model = model.to(device)
optim = torch.optim.Adam(lr=0.01, params=model.parameters())
criterion = nn.CrossEntropyLoss()
model.train()
for i, (img, c) in enumerate(dataloader):

    out = model(img)
    loss = criterion(out, c.view(1))

    optim.zero_grad()
    loss.backward()
    optim.step()

All advices are welcome. Thanks.


title: In Lstm choosing the hidden state

body:
if i choose
hidden_size = 3 #number of features in hidden state.
what does it mean in lstm. Why we need to use the hidden state features. What about choosing the size of the hidden state?


title: how to change RGBA img hue by gpu in python?

body:
As shown in the topic,i want to render animation in pyqt5,so it a part of plan,i already implement it in cpu version, it seems inefficient,becasue i want to refresh 30 frame per second at least,the numpy with cpu isn't satisfy my develop demand(seems slow),here is numpy version work on cpu code:
from PIL import Image
import numpy as np

def rgba_to_hsv(rgba):
    # Translated from source of colorsys.rgb_to_hsv
    # r,g,b should be a numpy arrays with values between 0 and 255
    # rgb_to_hsv returns an array of floats between 0.0 and 1.0.
    rgba = rgba.astype('float')
    hsv = np.zeros_like(rgba)
    # in case an RGBA array was passed, just copy the A channel
    # hsv[..., 3:] = rgba[..., 3:]
    # PrintXLS(rgba,'./data.xls')
    r, g, b ,a= rgba[..., 0], rgba[..., 1], rgba[..., 2],rgba[..., 3]
    maxc = np.max(rgba[..., :3], axis=-1)
    minc = np.min(rgba[..., :3], axis=-1)
    hsv[..., 2] = maxc
    mask = maxc != minc
    hsv[mask, 1] = (maxc - minc)[mask] / maxc[mask]
    rc = np.zeros_like(r)
    gc = np.zeros_like(g)
    bc = np.zeros_like(b)
    rc[mask] = (maxc - r)[mask] / (maxc - minc)[mask]
    gc[mask] = (maxc - g)[mask] / (maxc - minc)[mask]
    bc[mask] = (maxc - b)[mask] / (maxc - minc)[mask]
    hsv[..., 0] = np.select(
        [r == maxc, g == maxc], [bc - gc, 2.0 + rc - bc], default=4.0 + gc - rc)
    hsv[..., 0] = (hsv[..., 0] / 6.0) % 1.0
    # PrintXLS(hsv,'./data.xls')
    hsv[..., 3] = a
    # PrintXLS(hsv,'./data.xls')
    return hsv

def hsv_to_rgba(hsv):
    # Translated from source of colorsys.hsv_to_rgba
    # h,s should be a numpy arrays with values between 0.0 and 1.0
    # v should be a numpy array with values between 0.0 and 255.0
    # hsv_to_rgb returns an array of uints between 0 and 255.
    # HSV=hsv[..., 3:]
    rgba = np.empty_like(hsv)
    # rgba[..., 4:] = hsv[..., 4:]
    # print(rgba)
    h, s, v,a = hsv[..., 0], hsv[..., 1], hsv[..., 2],hsv[..., 3],
    i = (h * 6.0).astype('uint8')
    f = (h * 6.0) - i
    p = v * (1.0 - s)
    q = v * (1.0 - s * f)
    t = v * (1.0 - s * (1.0 - f))
    i = i % 6
    conditions = [s == 0.0, i == 1, i == 2, i == 3, i == 4, i == 5]
    rgba[..., 0] = np.select(conditions, [v, q, p, p, t, v], default=v)
    rgba[..., 1] = np.select(conditions, [v, v, v, q, p, p], default=t)
    rgba[..., 2] = np.select(conditions, [v, p, t, v, v, q], default=p)
    rgba[..., 3] = a
    # rgba[..., 3] =
    # print(rgba)
    # PrintXLS(rgba,'./data.xls')
    return rgba.astype('uint8')

def AshiftHue(img,HUE):
    arr = np.array(img)
    hsv = rgba_to_hsv(arr)
    hsv[..., 0] = (hsv[..., 0]+HUE) % 1.0
    rgba = hsv_to_rgba(hsv)
    return Image.fromarray(rgba, 'RGBA')

and torchvision had parallel method torchvision.transforms.functional.adjust_hue() but it doesn't support alpha channel,so i want to modify or construct a method like title,any good idea? thanks a lot for every solution