title: Pytorch model Parameters changes in CPU and GPU

body:
I have created the model and save the weights using google colab. Now I have created a prediction script.
The prediction script contains the model class. I am trying to load the model weights using the following method-
Saving & Loading Model Across Devices
Save on GPU, Load on CPU
Save:
torch.save(model.state_dict(), PATH)

Load:
device = torch.device('cpu')
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH, map_location=device))

The above method should work, right? Yes.
But when I am trying to do so I have different parameters of the model in Google Colab (Prediction, runtime-None, device=CPU) and different in my local machine (prediction, device=cpu)
Model Params in Colab-
def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')

The model has 12,490,234 trainable parameters

+-------------------------------------------------------+------------+
|                        Modules                        | Parameters |
+-------------------------------------------------------+------------+
|              encoder.tok_embedding.weight             |  2053376   |
|              encoder.pos_embedding.weight             |   25600    |
|      encoder.layers.0.self_attn_layer_norm.weight     |    256     |
|       encoder.layers.0.self_attn_layer_norm.bias      |    256     |
|         encoder.layers.0.ff_layer_norm.weight         |    256     |
|          encoder.layers.0.ff_layer_norm.bias          |    256     |
|      encoder.layers.0.self_attention.fc_q.weight      |   65536    |
|       encoder.layers.0.self_attention.fc_q.bias       |    256     |
|      encoder.layers.0.self_attention.fc_k.weight      |   65536    |
|       encoder.layers.0.self_attention.fc_k.bias       |    256     |
|      encoder.layers.0.self_attention.fc_v.weight      |   65536    |
|       encoder.layers.0.self_attention.fc_v.bias       |    256     |
|      encoder.layers.0.self_attention.fc_o.weight      |   65536    |
|       encoder.layers.0.self_attention.fc_o.bias       |    256     |
| encoder.layers.0.positionwise_feedforward.fc_1.weight |   131072   |
|  encoder.layers.0.positionwise_feedforward.fc_1.bias  |    512     |
| encoder.layers.0.positionwise_feedforward.fc_2.weight |   131072   |
|  encoder.layers.0.positionwise_feedforward.fc_2.bias  |    256     |
|      encoder.layers.1.self_attn_layer_norm.weight     |    256     |
|       encoder.layers.1.self_attn_layer_norm.bias      |    256     |
|         encoder.layers.1.ff_layer_norm.weight         |    256     |
|          encoder.layers.1.ff_layer_norm.bias          |    256     |
|      encoder.layers.1.self_attention.fc_q.weight      |   65536    |
|       encoder.layers.1.self_attention.fc_q.bias       |    256     |
|      encoder.layers.1.self_attention.fc_k.weight      |   65536    |
|       encoder.layers.1.self_attention.fc_k.bias       |    256     |
|      encoder.layers.1.self_attention.fc_v.weight      |   65536    |
|       encoder.layers.1.self_attention.fc_v.bias       |    256     |
|      encoder.layers.1.self_attention.fc_o.weight      |   65536    |
|       encoder.layers.1.self_attention.fc_o.bias       |    256     |
| encoder.layers.1.positionwise_feedforward.fc_1.weight |   131072   |
|  encoder.layers.1.positionwise_feedforward.fc_1.bias  |    512     |
| encoder.layers.1.positionwise_feedforward.fc_2.weight |   131072   |
|  encoder.layers.1.positionwise_feedforward.fc_2.bias  |    256     |
|      encoder.layers.2.self_attn_layer_norm.weight     |    256     |
|       encoder.layers.2.self_attn_layer_norm.bias      |    256     |
|         encoder.layers.2.ff_layer_norm.weight         |    256     |
|          encoder.layers.2.ff_layer_norm.bias          |    256     |
|      encoder.layers.2.self_attention.fc_q.weight      |   65536    |
|       encoder.layers.2.self_attention.fc_q.bias       |    256     |
|      encoder.layers.2.self_attention.fc_k.weight      |   65536    |
|       encoder.layers.2.self_attention.fc_k.bias       |    256     |
|      encoder.layers.2.self_attention.fc_v.weight      |   65536    |
|       encoder.layers.2.self_attention.fc_v.bias       |    256     |
|      encoder.layers.2.self_attention.fc_o.weight      |   65536    |
|       encoder.layers.2.self_attention.fc_o.bias       |    256     |
| encoder.layers.2.positionwise_feedforward.fc_1.weight |   131072   |
|  encoder.layers.2.positionwise_feedforward.fc_1.bias  |    512     |
| encoder.layers.2.positionwise_feedforward.fc_2.weight |   131072   |
|  encoder.layers.2.positionwise_feedforward.fc_2.bias  |    256     |
|              decoder.tok_embedding.weight             |  3209728   |
|              decoder.pos_embedding.weight             |   25600    |
|      decoder.layers.0.self_attn_layer_norm.weight     |    256     |
|       decoder.layers.0.self_attn_layer_norm.bias      |    256     |
|      decoder.layers.0.enc_attn_layer_norm.weight      |    256     |
|       decoder.layers.0.enc_attn_layer_norm.bias       |    256     |
|         decoder.layers.0.ff_layer_norm.weight         |    256     |
|          decoder.layers.0.ff_layer_norm.bias          |    256     |
|      decoder.layers.0.self_attention.fc_q.weight      |   65536    |
|       decoder.layers.0.self_attention.fc_q.bias       |    256     |
|      decoder.layers.0.self_attention.fc_k.weight      |   65536    |
|       decoder.layers.0.self_attention.fc_k.bias       |    256     |
|      decoder.layers.0.self_attention.fc_v.weight      |   65536    |
|       decoder.layers.0.self_attention.fc_v.bias       |    256     |
|      decoder.layers.0.self_attention.fc_o.weight      |   65536    |
|       decoder.layers.0.self_attention.fc_o.bias       |    256     |
|     decoder.layers.0.encoder_attention.fc_q.weight    |   65536    |
|      decoder.layers.0.encoder_attention.fc_q.bias     |    256     |
|     decoder.layers.0.encoder_attention.fc_k.weight    |   65536    |
|      decoder.layers.0.encoder_attention.fc_k.bias     |    256     |
|     decoder.layers.0.encoder_attention.fc_v.weight    |   65536    |
|      decoder.layers.0.encoder_attention.fc_v.bias     |    256     |
|     decoder.layers.0.encoder_attention.fc_o.weight    |   65536    |
|      decoder.layers.0.encoder_attention.fc_o.bias     |    256     |
| decoder.layers.0.positionwise_feedforward.fc_1.weight |   131072   |
|  decoder.layers.0.positionwise_feedforward.fc_1.bias  |    512     |
| decoder.layers.0.positionwise_feedforward.fc_2.weight |   131072   |
|  decoder.layers.0.positionwise_feedforward.fc_2.bias  |    256     |
|      decoder.layers.1.self_attn_layer_norm.weight     |    256     |
|       decoder.layers.1.self_attn_layer_norm.bias      |    256     |
|      decoder.layers.1.enc_attn_layer_norm.weight      |    256     |
|       decoder.layers.1.enc_attn_layer_norm.bias       |    256     |
|         decoder.layers.1.ff_layer_norm.weight         |    256     |
|          decoder.layers.1.ff_layer_norm.bias          |    256     |
|      decoder.layers.1.self_attention.fc_q.weight      |   65536    |
|       decoder.layers.1.self_attention.fc_q.bias       |    256     |
|      decoder.layers.1.self_attention.fc_k.weight      |   65536    |
|       decoder.layers.1.self_attention.fc_k.bias       |    256     |
|      decoder.layers.1.self_attention.fc_v.weight      |   65536    |
|       decoder.layers.1.self_attention.fc_v.bias       |    256     |
|      decoder.layers.1.self_attention.fc_o.weight      |   65536    |
|       decoder.layers.1.self_attention.fc_o.bias       |    256     |
|     decoder.layers.1.encoder_attention.fc_q.weight    |   65536    |
|      decoder.layers.1.encoder_attention.fc_q.bias     |    256     |
|     decoder.layers.1.encoder_attention.fc_k.weight    |   65536    |
|      decoder.layers.1.encoder_attention.fc_k.bias     |    256     |
|     decoder.layers.1.encoder_attention.fc_v.weight    |   65536    |
|      decoder.layers.1.encoder_attention.fc_v.bias     |    256     |
|     decoder.layers.1.encoder_attention.fc_o.weight    |   65536    |
|      decoder.layers.1.encoder_attention.fc_o.bias     |    256     |
| decoder.layers.1.positionwise_feedforward.fc_1.weight |   131072   |
|  decoder.layers.1.positionwise_feedforward.fc_1.bias  |    512     |
| decoder.layers.1.positionwise_feedforward.fc_2.weight |   131072   |
|  decoder.layers.1.positionwise_feedforward.fc_2.bias  |    256     |
|      decoder.layers.2.self_attn_layer_norm.weight     |    256     |
|       decoder.layers.2.self_attn_layer_norm.bias      |    256     |
|      decoder.layers.2.enc_attn_layer_norm.weight      |    256     |
|       decoder.layers.2.enc_attn_layer_norm.bias       |    256     |
|         decoder.layers.2.ff_layer_norm.weight         |    256     |
|          decoder.layers.2.ff_layer_norm.bias          |    256     |
|      decoder.layers.2.self_attention.fc_q.weight      |   65536    |
|       decoder.layers.2.self_attention.fc_q.bias       |    256     |
|      decoder.layers.2.self_attention.fc_k.weight      |   65536    |
|       decoder.layers.2.self_attention.fc_k.bias       |    256     |
|      decoder.layers.2.self_attention.fc_v.weight      |   65536    |
|       decoder.layers.2.self_attention.fc_v.bias       |    256     |
|      decoder.layers.2.self_attention.fc_o.weight      |   65536    |
|       decoder.layers.2.self_attention.fc_o.bias       |    256     |
|     decoder.layers.2.encoder_attention.fc_q.weight    |   65536    |
|      decoder.layers.2.encoder_attention.fc_q.bias     |    256     |
|     decoder.layers.2.encoder_attention.fc_k.weight    |   65536    |
|      decoder.layers.2.encoder_attention.fc_k.bias     |    256     |
|     decoder.layers.2.encoder_attention.fc_v.weight    |   65536    |
|      decoder.layers.2.encoder_attention.fc_v.bias     |    256     |
|     decoder.layers.2.encoder_attention.fc_o.weight    |   65536    |
|      decoder.layers.2.encoder_attention.fc_o.bias     |    256     |
| decoder.layers.2.positionwise_feedforward.fc_1.weight |   131072   |
|  decoder.layers.2.positionwise_feedforward.fc_1.bias  |    512     |
| decoder.layers.2.positionwise_feedforward.fc_2.weight |   131072   |
|  decoder.layers.2.positionwise_feedforward.fc_2.bias  |    256     |
|                 decoder.fc_out.weight                 |  3209728   |
|                  decoder.fc_out.bias                  |   12538    |
+-------------------------------------------------------+------------+
Total Trainable Params: 12490234

Model Params in Local-
def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')

The model has 12,506,137 trainable parameters

+-------------------------------------------------------+------------+
|                        Modules                        | Parameters |
+-------------------------------------------------------+------------+
|              encoder.tok_embedding.weight             |  2053376   |
|              encoder.pos_embedding.weight             |   25600    |
|      encoder.layers.0.self_attn_layer_norm.weight     |    256     |
|       encoder.layers.0.self_attn_layer_norm.bias      |    256     |
|         encoder.layers.0.ff_layer_norm.weight         |    256     |
|          encoder.layers.0.ff_layer_norm.bias          |    256     |
|      encoder.layers.0.self_attention.fc_q.weight      |   65536    |
|       encoder.layers.0.self_attention.fc_q.bias       |    256     |
|      encoder.layers.0.self_attention.fc_k.weight      |   65536    |
|       encoder.layers.0.self_attention.fc_k.bias       |    256     |
|      encoder.layers.0.self_attention.fc_v.weight      |   65536    |
|       encoder.layers.0.self_attention.fc_v.bias       |    256     |
|      encoder.layers.0.self_attention.fc_o.weight      |   65536    |
|       encoder.layers.0.self_attention.fc_o.bias       |    256     |
| encoder.layers.0.positionwise_feedforward.fc_1.weight |   131072   |
|  encoder.layers.0.positionwise_feedforward.fc_1.bias  |    512     |
| encoder.layers.0.positionwise_feedforward.fc_2.weight |   131072   |
|  encoder.layers.0.positionwise_feedforward.fc_2.bias  |    256     |
|      encoder.layers.1.self_attn_layer_norm.weight     |    256     |
|       encoder.layers.1.self_attn_layer_norm.bias      |    256     |
|         encoder.layers.1.ff_layer_norm.weight         |    256     |
|          encoder.layers.1.ff_layer_norm.bias          |    256     |
|      encoder.layers.1.self_attention.fc_q.weight      |   65536    |
|       encoder.layers.1.self_attention.fc_q.bias       |    256     |
|      encoder.layers.1.self_attention.fc_k.weight      |   65536    |
|       encoder.layers.1.self_attention.fc_k.bias       |    256     |
|      encoder.layers.1.self_attention.fc_v.weight      |   65536    |
|       encoder.layers.1.self_attention.fc_v.bias       |    256     |
|      encoder.layers.1.self_attention.fc_o.weight      |   65536    |
|       encoder.layers.1.self_attention.fc_o.bias       |    256     |
| encoder.layers.1.positionwise_feedforward.fc_1.weight |   131072   |
|  encoder.layers.1.positionwise_feedforward.fc_1.bias  |    512     |
| encoder.layers.1.positionwise_feedforward.fc_2.weight |   131072   |
|  encoder.layers.1.positionwise_feedforward.fc_2.bias  |    256     |
|      encoder.layers.2.self_attn_layer_norm.weight     |    256     |
|       encoder.layers.2.self_attn_layer_norm.bias      |    256     |
|         encoder.layers.2.ff_layer_norm.weight         |    256     |
|          encoder.layers.2.ff_layer_norm.bias          |    256     |
|      encoder.layers.2.self_attention.fc_q.weight      |   65536    |
|       encoder.layers.2.self_attention.fc_q.bias       |    256     |
|      encoder.layers.2.self_attention.fc_k.weight      |   65536    |
|       encoder.layers.2.self_attention.fc_k.bias       |    256     |
|      encoder.layers.2.self_attention.fc_v.weight      |   65536    |
|       encoder.layers.2.self_attention.fc_v.bias       |    256     |
|      encoder.layers.2.self_attention.fc_o.weight      |   65536    |
|       encoder.layers.2.self_attention.fc_o.bias       |    256     |
| encoder.layers.2.positionwise_feedforward.fc_1.weight |   131072   |
|  encoder.layers.2.positionwise_feedforward.fc_1.bias  |    512     |
| encoder.layers.2.positionwise_feedforward.fc_2.weight |   131072   |
|  encoder.layers.2.positionwise_feedforward.fc_2.bias  |    256     |
|              decoder.tok_embedding.weight             |  3217664   |
|              decoder.pos_embedding.weight             |   25600    |
|      decoder.layers.0.self_attn_layer_norm.weight     |    256     |
|       decoder.layers.0.self_attn_layer_norm.bias      |    256     |
|      decoder.layers.0.enc_attn_layer_norm.weight      |    256     |
|       decoder.layers.0.enc_attn_layer_norm.bias       |    256     |
|         decoder.layers.0.ff_layer_norm.weight         |    256     |
|          decoder.layers.0.ff_layer_norm.bias          |    256     |
|      decoder.layers.0.self_attention.fc_q.weight      |   65536    |
|       decoder.layers.0.self_attention.fc_q.bias       |    256     |
|      decoder.layers.0.self_attention.fc_k.weight      |   65536    |
|       decoder.layers.0.self_attention.fc_k.bias       |    256     |
|      decoder.layers.0.self_attention.fc_v.weight      |   65536    |
|       decoder.layers.0.self_attention.fc_v.bias       |    256     |
|      decoder.layers.0.self_attention.fc_o.weight      |   65536    |
|       decoder.layers.0.self_attention.fc_o.bias       |    256     |
|     decoder.layers.0.encoder_attention.fc_q.weight    |   65536    |
|      decoder.layers.0.encoder_attention.fc_q.bias     |    256     |
|     decoder.layers.0.encoder_attention.fc_k.weight    |   65536    |
|      decoder.layers.0.encoder_attention.fc_k.bias     |    256     |
|     decoder.layers.0.encoder_attention.fc_v.weight    |   65536    |
|      decoder.layers.0.encoder_attention.fc_v.bias     |    256     |
|     decoder.layers.0.encoder_attention.fc_o.weight    |   65536    |
|      decoder.layers.0.encoder_attention.fc_o.bias     |    256     |
| decoder.layers.0.positionwise_feedforward.fc_1.weight |   131072   |
|  decoder.layers.0.positionwise_feedforward.fc_1.bias  |    512     |
| decoder.layers.0.positionwise_feedforward.fc_2.weight |   131072   |
|  decoder.layers.0.positionwise_feedforward.fc_2.bias  |    256     |
|      decoder.layers.1.self_attn_layer_norm.weight     |    256     |
|       decoder.layers.1.self_attn_layer_norm.bias      |    256     |
|      decoder.layers.1.enc_attn_layer_norm.weight      |    256     |
|       decoder.layers.1.enc_attn_layer_norm.bias       |    256     |
|         decoder.layers.1.ff_layer_norm.weight         |    256     |
|          decoder.layers.1.ff_layer_norm.bias          |    256     |
|      decoder.layers.1.self_attention.fc_q.weight      |   65536    |
|       decoder.layers.1.self_attention.fc_q.bias       |    256     |
|      decoder.layers.1.self_attention.fc_k.weight      |   65536    |
|       decoder.layers.1.self_attention.fc_k.bias       |    256     |
|      decoder.layers.1.self_attention.fc_v.weight      |   65536    |
|       decoder.layers.1.self_attention.fc_v.bias       |    256     |
|      decoder.layers.1.self_attention.fc_o.weight      |   65536    |
|       decoder.layers.1.self_attention.fc_o.bias       |    256     |
|     decoder.layers.1.encoder_attention.fc_q.weight    |   65536    |
|      decoder.layers.1.encoder_attention.fc_q.bias     |    256     |
|     decoder.layers.1.encoder_attention.fc_k.weight    |   65536    |
|      decoder.layers.1.encoder_attention.fc_k.bias     |    256     |
|     decoder.layers.1.encoder_attention.fc_v.weight    |   65536    |
|      decoder.layers.1.encoder_attention.fc_v.bias     |    256     |
|     decoder.layers.1.encoder_attention.fc_o.weight    |   65536    |
|      decoder.layers.1.encoder_attention.fc_o.bias     |    256     |
| decoder.layers.1.positionwise_feedforward.fc_1.weight |   131072   |
|  decoder.layers.1.positionwise_feedforward.fc_1.bias  |    512     |
| decoder.layers.1.positionwise_feedforward.fc_2.weight |   131072   |
|  decoder.layers.1.positionwise_feedforward.fc_2.bias  |    256     |
|      decoder.layers.2.self_attn_layer_norm.weight     |    256     |
|       decoder.layers.2.self_attn_layer_norm.bias      |    256     |
|      decoder.layers.2.enc_attn_layer_norm.weight      |    256     |
|       decoder.layers.2.enc_attn_layer_norm.bias       |    256     |
|         decoder.layers.2.ff_layer_norm.weight         |    256     |
|          decoder.layers.2.ff_layer_norm.bias          |    256     |
|      decoder.layers.2.self_attention.fc_q.weight      |   65536    |
|       decoder.layers.2.self_attention.fc_q.bias       |    256     |
|      decoder.layers.2.self_attention.fc_k.weight      |   65536    |
|       decoder.layers.2.self_attention.fc_k.bias       |    256     |
|      decoder.layers.2.self_attention.fc_v.weight      |   65536    |
|       decoder.layers.2.self_attention.fc_v.bias       |    256     |
|      decoder.layers.2.self_attention.fc_o.weight      |   65536    |
|       decoder.layers.2.self_attention.fc_o.bias       |    256     |
|     decoder.layers.2.encoder_attention.fc_q.weight    |   65536    |
|      decoder.layers.2.encoder_attention.fc_q.bias     |    256     |
|     decoder.layers.2.encoder_attention.fc_k.weight    |   65536    |
|      decoder.layers.2.encoder_attention.fc_k.bias     |    256     |
|     decoder.layers.2.encoder_attention.fc_v.weight    |   65536    |
|      decoder.layers.2.encoder_attention.fc_v.bias     |    256     |
|     decoder.layers.2.encoder_attention.fc_o.weight    |   65536    |
|      decoder.layers.2.encoder_attention.fc_o.bias     |    256     |
| decoder.layers.2.positionwise_feedforward.fc_1.weight |   131072   |
|  decoder.layers.2.positionwise_feedforward.fc_1.bias  |    512     |
| decoder.layers.2.positionwise_feedforward.fc_2.weight |   131072   |
|  decoder.layers.2.positionwise_feedforward.fc_2.bias  |    256     |
|                 decoder.fc_out.weight                 |  3217664   |
|                  decoder.fc_out.bias                  |   12569    |
+-------------------------------------------------------+------------+
Total Trainable Params: 12506137

So, that's why I am unable to load the model. Because the model has a different parameter in local.
Even if I try to load the weights in local it gives me-
model.load_state_dict(torch.load(f"{model_name}.pt", map_location=device))

Error-
--------------------------------------------------------------------------- RuntimeError                              Traceback (most recent call last) <ipython-input-24-f5baac4441a5> in <module>
----> 1 model.load_state_dict(torch.load(f"{model_name}_2.pt", map_location=device))

c:\anaconda\envs\lang_trans\lib\site-packages\torch\nn\modules\module.py in load_state_dict(self, state_dict, strict)
    845         if len(error_msgs) > 0:
    846             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 847                                self.__class__.__name__, "\n\t".join(error_msgs)))
    848         return _IncompatibleKeys(missing_keys, unexpected_keys)
    849

RuntimeError: Error(s) in loading state_dict for Seq2Seq:   size mismatch for decoder.tok_embedding.weight: copying a param with shape torch.Size([12538, 256]) from checkpoint, the shape in current model is torch.Size([12569, 256]).    size mismatch for decoder.fc_out.weight: copying a param with shape torch.Size([12538, 256]) from checkpoint, the shape in current model is torch.Size([12569, 256]).   size mismatch for decoder.fc_out.bias: copying a param with shape torch.Size([12538]) from checkpoint, the shape in current model is torch.Size([12569]).

The model param of the local must be wrong because in colab (device=CPU, runtime=None) I am able to load the weights after defining model class. But in the local machine the params changes, so I am unable to load the weights. I know it's weird, help me to find the solution.
You can check the full code of the model here-

<script src="https://gist.github.com/Dipeshpal/90c715a7b7f00845e20ef998bda35835.js"></script>

https://gist.github.com/Dipeshpal/90c715a7b7f00845e20ef998bda35835
After this model params change.


title: Pytorch Custom data loading with HTTPS is very slow

body:
I tried implementing a custom data loader that will make a web request and will return a sample. My purpose of the program is to see if this idea would be faster than the original data loader. My web server code is run with
srun -n24 --mem = 12g python web.py

Which will then create 24 "workers" that run in the cluster. Then each worker will write its portname to a file to make itself known to the data loader that he exist. So, when the dataloader is called in the training loop. The data loader selects a random server from the files and send them a web request with an index. The web server will then load the sample and do augmentation and return via http response. From my view, i thought it would be faster than the original data loader as, each data loader worker would send a request to the webserver and get a sample. Thus, distributing data to different server so they load the images faster.
However, when i do a comparison with original data using COCO dataset. The original data loader takes 743.820 sec to complete loading an epoch while my custom data loader takes 1503.26 sec to complete. I couldn't figure out which part of my code is taking a long time, so i would like to ask for assistance. Please if my explaination is bad/not great please let me know. Any help is appreciated. Thankyou.
The following the code for starting webserver:
class PytorchDataHandler(BaseHTTPRequestHandler):
    def do_GET(self):

        self.send_response(200)

        self.end_headers()

        get_param = self.path
        get_param = parse_qs(urlparse(get_param).query)
        batch_list = [[],[]]
        c_batches = []
        index = get_param['index']
        if index :
            for data in index:
                result = imagenet_data[int(data)]
                batch_list[0].append(result[0])
                batch_list[1].append(result[1])
            c_batches.append(batch_list)
            torch.save(batch_list, self.wfile)

        else:
            write_log('Empty Parameter')

def main():
    sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
    hostname = socket.gethostname()
    n_hostname = hostname.split(".")

    # Bind to random port
    sock.bind(('0.0.0.0', 0))
    # Get Port Number
    PORT = int(sock.getsockname()[1])

    current_dir = os.getcwd()
    create_dir = os.path.join(current_dir, r'worker_file')

    #filename = create_dir + '/' +  str(n_hostname[0]) + '.cvl-tengig:' + str(PORT)
    filename = create_dir + '/' +  str(n_hostname[0])  + ':' + str(PORT)
    os.makedirs(os.path.dirname(filename), exist_ok=True)

    open_file = open(filename, 'w')
    open_file.write(str(n_hostname[0]) + ':' + str(PORT))

    open_file.close()
    try :
        SERVER = HTTPServer(('', PORT), PytorchDataHandler)
        SERVER.serve_forever()
    except KeyboardInterrupt:
        print('Shutting down server, ^C')
        os.remove(filename)
        SERVER.socket.close()

if __name__ == '__main__':
    main()

The code for custom data loader:
class DistData(Dataset):
    def __init__(self, data, transform = None):
        self.data = data
        # Get file path
        current_dir = os.getcwd()
        create_dir = os.path.join(current_dir, r'worker_file')

        # Get all item in file
        self.arr = os.listdir(create_dir)
        self.selected = []

    def __getitem__(self, index):
        # Select a random server

        random_server = random.choice(self.arr)

        # Remove selected server from the server list
        self.arr.remove(random_server)

        # Append selected server to the selected list
        self.selected.append(random_server)

        return self.post_request(index, random_server)

    def __len__(self):
        return len(self.data)

    def post_request(self, index, random_server):
        params = {'index': index}
        url = 'http://' + random_server + '/get'

        r = requests.get(url , params = params)

        print("Response Time : {:<10} , worker : {:<10} ".format(r.elapsed.total_seconds(), torch.utils.data.get_worker_info().id ))

        # Remove server from selected once there's a response
        self.selected.remove(random_server)
        # Add back to main server list after response
        self.arr.append(random_server)

        buffer = io.BytesIO(r.content)
        response = torch.load(buffer)

        return response

def train(net, device, trainloader, criterion, optimizer):
    for epoch in range(2):  # loop over the dataset multiple times
        running_loss = 0.0
        print('Epoch : {}'.format(epoch + 1))
        print('----------------------------')
        start_time = time.time()
        total_time = 0
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data
            print("Train: Time taken to load batch {} is {}".format(i+1,time.time() - start_time))
            total_time += time.time() - start_time
            start_time = time.time()
        print('Epoch : {} , Total Time Taken : {}'.format(epoch + 1, total_time))
    print('Finished Training')

    imagenet_data =torchvision.datasets.CocoCaptions('/db/shared/detection+classification/coco/train2017/' ,
                            '/db/shared/detection+classification/coco/annotations/captions_train2017.json')

    training_set = DistData(imagenet_data)
    trainloader = DataLoader(training_set, sampler = BatchSampler(RandomSampler(training_set), batch_size = 24, drop_last = False),
                num_workers = 4)

train(trainloader)


title: How can I repartition RDD by key and then pack it to shards?

body:
I have many files containing millions of rows in format:
id, created_date, some_value_a, some_value_b, some_value_c

This way of repartitioning was super slow and created for me over million of small ~500b files:
    rdd_df = rdd.toDF(["id", "created_time", "a", "b", "c"])
    rdd_df.write.partitionBy("id").csv("output")

I would like to achieve output files, where each file contains like 10000 unique IDs and all their rows.
How could I achieve something like this?


title: How to set a separate learnable learning rate for each parameter in Pytorch?

body:
I am a little bit confused about the learnable learning rate for each parameter in Pytorch.
I want to set a separate learnable learning rate for each parameter. The learning rate and model parameters will be optimized together during the training process.
I find some code on stackoverFlow：
    optimizer = optim.Adam([
            {'params': net.layer1.weight},
            {'params': net.layer1.bias, 'lr': 0.01},
            {'params': net.layer2.weight, 'lr': 0.001}
        ], lr=0.1, weight_decay=0.0001)

This method solves the problem of setting a fixed learning rate for each parameter, but it cannot allow the learning rate to be trained along with the model.
I tried to define a class with learnable learning rate:
    class Learner(nn.Module):

        def __init__(self, net, in_channels, num_classes):
            super(Learner, self).__init__()
            self.learner = net(in_channels, num_classes)
            self.task_lr = OrderedDict()

        def forward(self, X):
            out = self.learner(X)
            return out

        def define_lr(self):
            for key, val in self.named_parameters():
                self.task_lr[key] = nn.Parameter(
                    1e-4 * torch.ones_like(val, requires_grad=True))

and then:
    model_params = list(model.parameters()) + list(model.task_lr.values())
    optimizer = optim.Adam(model_params, lr=args.lr)
    adapted_state_dict = model.cloned_state_dict()

    #####compute loss#####
    #......
    ######################

    model.zero_grads()
    loss.backward()
    for name, param in model.named_parameters():
        if param.grad is not None:
            adapted_state_dict[name] -= model.task_lr[name] * param.grad.data
    model.load_state_dict(adapted_state_dict)
    optimizer.step()

But this method still cannot train the learning rate.
Feel free to ask if more code is needed to explain the problem.


title: while running netron on colab, getting this "OSError: [Errno 98] Address already in use" error

body:
I'm using Netron, for visualizing the model on Colab. as shown in this notebook line 11.
when I run the following script to view the model,
import netron

enable_netron = True
if enable_netron:
    netron.start(optimized_model_path)

am getting this error:
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-13-86b0c5c2423c> in <module>()
      5 if enable_netron:
----> 6     netron.start(optimized_model_path)

5 frames
/usr/lib/python3.6/socketserver.py in server_bind(self)
    468         if self.allow_reuse_address:
    469             self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
--> 470         self.socket.bind(self.server_address)
    471         self.server_address = self.socket.getsockname()
    472

OSError: [Errno 98] Address already in use

how can I fix this issue?
Ik I can use its desktop version of the app and give it the model but how can use Netron in colab?


title: What does the 1 in torch.Size([64, 1, 28, 28]) mean when I check a tensor shape?

body:
I'm following this tutorial on towardsdatascience.com because I wanted to try the MNIST dataset using Pytorch since I've already done it using keras.
So in Step 2, knowing the dataset better, they print the trainloader's shape and it returns torch.Size([64, 1, 28, 28]). I understand  that 64 is the number of images in that loader and that each one is a 28x28 image but what does the 1 mean exactly?


title: How to look at the parameters of a pytorch model?

body:
I have a simple pytorch neural net that I copied from openai, and I modified it to some extent (mostly the input).

When I run my code, the output of the network remains the same on every episode, as if no training occurs.

I want to see if any training happens, or if some other reason causes the results to be the same.

How can I make sure any movement happens to the weights?

Thanks


title: RuntimeError: size mismatch, m1: [28 x 28], m2: [784 x 128]

body:
After training my model, I tried to plot graph of the softmax output, but it resulted in the runtime error mentioned in the title.

Here is the following code snippet:

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper

# Test out your network!

dataiter = iter(testloader)
images, labels = dataiter.next()
img = images[1]

# TODO: Calculate the class probabilities (softmax) for img
ps = torch.exp(model(img))

# Plot the image and probabilities
helper.view_classify(img, ps, version='Fashion')


title: How do I update a tensor in Pytorch after indexing twice?

body:
I know how to update a tensor after indexing into part of it like this:

import torch

b = torch.tensor([0, 1, 0, 1], dtype=torch.uint8)
b[b] = 2
b
# tensor([0, 2, 0, 2], dtype=torch.uint8)

but is there a way I can update the original tensor after indexing into it twice? E.g.

i = 1
b = torch.tensor([0, 1, 0, 1], dtype=torch.uint8)
b[b][i] = 2
b
# tensor([0, 1, 0, 1], dtype=torch.uint8)

What I'd like is for b to be tensor([0, 1, 0, 2]) at the end. Is there a way to do this?

I know that I can do

masked = b[b]
masked[i] = 2
b[b] = masked
b
# tensor([0, 1, 0, 2], dtype=torch.uint8)

but is there any better way? It seems that this must be inefficient; if masked is very large, I'm updating many locations in b when I've really only changed one.

(In case a different approach than indexing twice would work better, the general problem I have is how to change the value in an original tensor at the ith location of a masked version of that tensor.)


title: PyTorch: why is the loss unchanging, in this simple example?

body:
I'm writing a code example to do a simple linear projection (like PCA) in PyTorch. Everything appears to be OK except that the loss does not change as training progresses. Changing the learning rate doesn't affect this, and it's a simple one-dimensional problem so the loss should certainly be changing. What am I missing here?

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as nnF

class PCArot2D(nn.Module):
    "2D PCA rotation, expressed as a gradient-descent problem"
    def __init__(self):
        super(PCArot2D, self).__init__()
        self.theta = nn.Parameter(torch.tensor(np.random.random() * 2 * np.pi))

    def getrotation(self):
        sintheta = torch.sin(self.theta)
        costheta = torch.cos(self.theta)
        return torch.tensor([[costheta, -sintheta], [sintheta, costheta]], requires_grad=True, dtype=torch.double)

    def forward(self, x):
        xmeans = torch.mean(x, dim=1, keepdim=True)
        rot = self.getrotation()

        return torch.mm(rot, x - xmeans)

def covariance(y):
    "Calculates the covariance matrix of its input (as torch variables)"
    ymeans = torch.mean(y, dim=1, keepdim=True)
    ycentred = y - ymeans
    return torch.mm(ycentred, ycentred.T) / ycentred.shape[1]

net = PCArot2D()

example2 = torch.tensor(np.random.randn(2, 33))

# define a loss function and an optimiser
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.001, momentum=0.1)

# train the network
num_epochs = 1000
for epoch in range(num_epochs):
    optimizer.zero_grad()
    # forward + backward + optimize
    outputs = net(torch.DoubleTensor(example2))
    # the covariance between output channels is the measure that we wish to minimise
    covariance = (outputs[0, :] * outputs[1, :]).mean()
    loss = criterion(covariance, torch.tensor(0, dtype=torch.double))

    loss.backward()
    optimizer.step()

    running_loss = loss.item()
    if ((epoch & (epoch - 1)) == 0) or epoch==(num_epochs-1): # don't print on all epochs
        # print statistics
        print('[%d] loss: %.8f' %
            (epoch, running_loss))

print('Finished Training')

Output:

[0] loss: 0.00629047
[1] loss: 0.00629047
[2] loss: 0.00629047
[4] loss: 0.00629047
[8] loss: 0.00629047
etc


title: Normalize Tensor Image - Optimization

body:
Im using a torchmodel to do the inference on. While running the inference I noticed that the normalization of the tensor image consumes half the inference time.

The function that is called:

input_image_normalizer = Compose([
    ToTensor(),
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

When removing the normalize function, the inference FPS is doubled.

Is there any optimization that can be done to lower the time of the normalization operation? I'm thinking of putting the Normalize function on the GPU instead? If so how could that be done?
I have CUDA, and torchvision.

Normalize function: https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Normalize


title: Invalid syntax error when setting root = to in PyTorch

body:
import torchvision
from torchvision import transforms
train_data_path="./train/"
transforms = transforms.Compose([
transforms.Resize(64),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225] )
])
train_data = torchvision.datasets.ImageFolder
(root=train_data_path,transform=transforms)

This is the error message:
File "<ipython-input-4-e470172b3902>", line 8
    (root=train_data_path,transform=transforms)
         ^
SyntaxError: invalid syntax

How would I be able to fix this?


title: Deploy pytorch model in pyspark failed

body:
I am running the inference of pytorch model over pyspark. when I use a simple model on test data blow it works:
%spark2_1.pyspark

import torch
import torch.nn as nn
# from pyspark.sql.functions import col, pandas_udf, PandasUDFType, udf
import pyspark.sql.functions as F
from pyspark.sql import SparkSession
from pyspark.sql.types import ArrayType, FloatType, DoubleType
import pandas as pd
import numpy as np

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.w = nn.Linear(5, 1)

    def forward(self, x):
        return self.w(x)

net = Net()
bc_model_state = sc.broadcast(net.state_dict())

def sum_cols(a,b,c,d):
    return a+b+c+d

sum_cols_udf = udf(sum_cols, FloatType())

df = sc.parallelize([[np.random.rand() for i in range(5)] for j in range(5000000)]).toDF()
df = df.withColumn('features', F.array([F.col(f"_{i}") for i in range(1, 6)])).repartition(20000)
df = df.withColumn('sum_cols', sum_cols(col('_1'),col('_2'),col('_3'),col('_4')))

def get_model_for_eval():
  # Broadcast the model state_dict
  net.load_state_dict(bc_model_state.value)
  net.eval()
  return net

def one_row_predict(x):
    model = get_model_for_eval()
    t = torch.tensor(x, dtype=torch.float32)
    prediction = model(t).cpu().detach().item()
    return prediction

one_row_udf = udf(one_row_predict, FloatType())
df = df.withColumn('pred_one_row', one_row_udf(col('features')))
df.show(10)

+--------------------+-------------------+-------------------+-------------------+-------------------+--------------------+------------------+-------------+
|                  _1|                 _2|                 _3|                 _4|                 _5|            features|          sum_cols| pred_one_row|
+--------------------+-------------------+-------------------+-------------------+-------------------+--------------------+------------------+-------------+
|   0.988154756017273|   0.99582324295928|0.23366768659825665|0.12221273947098088| 0.9166397110611093|[0.98815475601727...|  2.33985842504579|   0.14895654|
|3.659617010587368E-4|0.16142802396028821| 0.6799687123705221| 0.4893824104510843| 0.9168979701228083|[3.65961701058736...|1.3311451084829535|   0.57212985|
| 0.09900675712724982|0.34704430521398266|0.02799883775589418| 0.7017933079205884| 0.6746628345467317|[0.09900675712724...| 1.175843208017715|   0.23441058|
| 0.12929918963939846| 0.6805534130783143|0.43567158419110397| 0.8943169409241903|0.11256967960674524|[0.12929918963939...| 2.139841127833007|-0.0075862706|
|  0.5472075668098001| 0.9461527823407179| 0.8440631308714541|0.09589382600049867|0.48732081612145683|[0.54720756680980...|  2.43331730602247|   0.12818295|
|  0.3425360932298782| 0.5592018009995154| 0.8206231183620638| 0.2553154387123161| 0.5699215561391294|[0.34253609322987...|1.9776764513037737|    0.3091246|
|   0.951280354909562|0.20552269503212284|0.48470974551843626| 0.5559824288843389| 0.5433303631041886|[0.95128035490956...|  2.19749522434446|   0.40010738|
| 0.14774825217619647| 0.6607511689023305| 0.5594334053077861| 0.4285903280218679| 0.7355948624449975|[0.14774825217619...| 1.796523154408181|   0.26837745|
|  0.0212829941901852| 0.7964859368528711|0.11005573018008041| 0.9770942505586772| 0.6699409258514414|[0.02128299419018...| 1.904918911781814|   0.09390697|
|  0.3521362144627097| 0.1145780732617927| 0.2493492040463351| 0.7615716255563724|0.26349784477095695|[0.35213621446270...|  1.47763511732721|   0.23092186|
+--------------------+-------------------+-------------------+-------------------+-------------------+--------------------+------------------+-------------+
only showing top 10 rows

But when I run a little complicate model on my data, it fails. Here is the data:
+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------+--------+-----------------+
|        global_media|      mediatypefirst|     mediatypesecond|                camp|            industry|        business_tag|        appname_list|    packagename_list|          brand_list|        product_list|            pkg_attr|city_idx|manu_idx|               fc|
+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------+--------+-----------------+
|[1.0, 0.0, 0.0, 0...|[1.0, 0.0, 0.0, 0...|[1.0, 0.0, 0.0, 0...|[0.0, 0.0, 0.0, 0...|[0.0, 0.0, 0.0, 0...|[1.0, 0.0, 0.0, 0...|[1.0, 0.0, 0.0, 0...|[1.0, 0.0, 0.0, 0...|[1.0, 0.0, 0.0, 0...|[1.0, 0.0, 0.0, 0...|[0.0, 1.0, 0.0, 0...|     2.0|     0.0|[1.0,1.0,0.0,5.0]|
|[1.0, 1.0, 1.0, 1...|[1.0, 1.0, 1.0, 0...|[1.0, 1.0, 1.0, 0...|[0.0, 1.0, 0.0, 1...|[1.0, 1.0, 1.0, 1...|[1.0, 0.0, 0.0, 0...|[0.0, 1.0, 1.0, 1...|[0.0, 1.0, 1.0, 0...|[0.0, 1.0, 1.0, 0...|[0.0, 1.0, 1.0, 1...|[1.0, 0.0, 1.0, 0...|  1448.0|     0.0|[1.0,0.0,1.0,3.0]|
+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------+--------+-----------------+
only showing top 2 rows

And the model:
%spark2_1.pyspark

import torch
from torch import nn, optim
import torch.nn.functional as F
from random import random

from sklearn.metrics import accuracy_score

class Item2vec(nn.Module):
    def __init__(self, glob_med_embSize=16, med_fir_embSize=3, med_sec_embSize=5, camp_embSize=30, industry_embSize=10,
                 bus_tag_embSize=5, app_embSize=10, pkg_embSize=10, brand_embSize=5, prod_embSize=5, pkg_attr_embSize=5,
                 city_embSize=30, manu_embSize=10):
        super(Item2vec, self).__init__()

        self.gm_emb = nn.Linear(cv_dict['global_media_cv'].shape[1], int(pow(cv_dict['global_media_cv'].shape[1], 1/3))+1)
        self.mf_emb = nn.Linear(cv_dict['mediatypefirst_cv'].shape[1], int(pow(cv_dict['mediatypefirst_cv'].shape[1], 1/3))+1)
        self.ms_emb = nn.Linear(cv_dict['mediatypesecond_cv'].shape[1], int(pow(cv_dict['mediatypesecond_cv'].shape[1], 1/3))+1)
        self.camp_emb = nn.Linear(cv_dict['camp_cv'].shape[1], int(pow(cv_dict['camp_cv'].shape[1], 1/3))+1)
        self.ind_emb = nn.Linear(cv_dict['industry_cv'].shape[1], int(pow(cv_dict['industry_cv'].shape[1], 1/3))+1)
        self.bt_emb = nn.Linear(cv_dict['business_tag_cv'].shape[1], int(pow(cv_dict['business_tag_cv'].shape[1], 1/3))+1)
        self.app_emb = nn.Linear(cv_dict['appname_list_cv'].shape[1], int(pow(cv_dict['appname_list_cv'].shape[1], 1/3))+1)
        self.pkg_emb = nn.Linear(cv_dict['packagename_list_cv'].shape[1], int(pow(cv_dict['packagename_list_cv'].shape[1], 1/3))+1)
        self.brand_emb = nn.Linear(cv_dict['brand_list_cv'].shape[1], int(pow(cv_dict['brand_list_cv'].shape[1], 1/3))+1)
        self.prod_emb = nn.Linear(cv_dict['product_list_cv'].shape[1], int(pow(cv_dict['product_list_cv'].shape[1], 1/3))+1)
        self.pkgattr_emb = nn.Linear(cv_dict['pkg_attr_cv'].shape[1], int(pow(cv_dict['pkg_attr_cv'].shape[1], 1/3))+1)

        self.city_emb = nn.Embedding(cv_dict['city'].max().item()+1, int(pow(cv_dict['city'].max().item(), 1/3))+1)
        self.manu_emb = nn.Embedding(cv_dict['manu'].max().item()+1, int(pow(cv_dict['manu'].max().item(), 1/3))+1)

        self.fc_emb =  nn.Linear(cv_dict['fc'].shape[1], 20)

        embed_dim = 0
        for i in ['global_media_cv', 'mediatypefirst_cv', 'mediatypesecond_cv', 'camp_cv', 'industry_cv', 'business_tag_cv', 'appname_list_cv',
                            'packagename_list_cv', 'brand_list_cv', 'product_list_cv', 'pkg_attr_cv']:
            embed_dim += (int(pow(cv_dict[i].shape[1], 1/3))+1)

        embed_dim += int(pow(cv_dict['city'].max().item(), 1/3))+1
        embed_dim += int(pow(cv_dict['manu'].max().item(), 1/3))+1
        embed_dim += 20

        self.fc1 = nn.Linear(embed_dim, 200)
        self.fc2 = nn.Linear(200, 200)
        self.out = nn.Linear(200, 2)

    def forward(self, gm, mf, ms, camp, ind, bt, app, pkg, brand, prod, pkgattr, city, manu, fc_data):
        gm_embs = self.gm_emb(gm)
        mf_embs = self.mf_emb(mf)
        ms_embs = self.ms_emb(ms)
        camp_embs = self.camp_emb(camp)
        ind_embs = self.ind_emb(ind)
        bt_embs = self.bt_emb(bt)
        app_embs = self.app_emb(app)
        pkg_embs = self.pkg_emb(pkg)
        brand_embs = self.brand_emb(brand)
        prod_embs = self.prod_emb(prod)
        pkg_attr_embs = self.pkgattr_emb(pkgattr)
        city_embs = self.city_emb(city)
        manu_embs = self.manu_emb(manu)
        fc_embs = self.fc_emb(fc_data)

        x = torch.cat([gm_embs,mf_embs,ms_embs,camp_embs,ind_embs,bt_embs,app_embs,pkg_embs,brand_embs,prod_embs,pkg_attr_embs,city_embs, manu_embs, fc_embs], dim=-1)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.out(x)

        return x

    def predict(self, gm, mf, ms, camp, ind, bt, app, pkg, brand, prod, pkgattr, city, manu, fc_data):
        pred = F.softmax(self.forward(gm, mf, ms, camp, ind, bt, app, pkg, brand, prod, pkgattr, city, manu, fc_data))
        probs = pred.detach().numpy()[:, 1]
        ans = []
        for t in pred:
            if t[0]>t[1]:
                ans.append(0)
            else:
                ans.append(1)
        return torch.tensor(ans), probs

The model was trained in local, and here is inference of the model over pyspark:
%spark2_1.pyspark

bc_model_state = sc.broadcast(model.state_dict())
net = Item2vec()

def get_model_for_eval_demo():
  # Broadcast the model state_dict
  net.load_state_dict(bc_model_state.value)
  net.eval()
  return net

def one_row_predict_demo(global_media,mediatypefirst,mediatypesecond,camp,industry,business_tag,appname_list,packagename_list,brand_list,product_list,pkg_attr,city_idx,manu_idx,fc):
    bc_model = get_model_for_eval_demo()
    # #bc_model.eval()
    # for row in partitions:
    gm = torch.tensor(global_media, dtype=torch.float32)
    mf = torch.tensor(mediatypefirst, dtype=torch.float32)
    ms = torch.tensor(mediatypesecond, dtype=torch.float32)
    camp = torch.tensor(camp, dtype=torch.float32)
    ind = torch.tensor(industry, dtype=torch.float32)
    bt = torch.tensor(business_tag, dtype=torch.float32)
    app = torch.tensor(appname_list, dtype=torch.float32)
    pkg = torch.tensor(packagename_list, dtype=torch.float32)
    brand = torch.tensor(brand_list, dtype=torch.float32)
    prod = torch.tensor(product_list, dtype=torch.float32)
    pkg_attr = torch.tensor(pkg_attr, dtype=torch.float32)
    city = torch.tensor(city_idx, dtype=torch.long)
    manu = torch.tensor(manu_idx, dtype=torch.long)
    fc = torch.tensor(fc, dtype=torch.float32)

    with torch.no_grad():
        _, prob = bc_model.predict(gm, mf, ms, camp, ind, bt, app, pkg, brand, prod, pkg_attr, city, manu, fc)

    # prob = bc_model(fc).cpu().detach().item()
    # yeild (row['id'], row.label, prob)
    return prob

one_row_predict_demo_udf = udf(one_row_predict_demo, DoubleType())
df = demo.repartition(100).withColumn('demo_prob', one_row_predict_demo_udf(col('global_media'),col('mediatypefirst'),col('mediatypesecond'),col('camp'),col('industry'),col('business_tag'),
            col('appname_list'),col('packagename_list'),col('brand_list'),col('product_list'),col('pkg_attr'),col('city_idx'),col('manu_idx'),col('fc')))
df.show(4)

Error message:
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in dump(self, obj)
    146         try:
--> 147             return Pickler.dump(self, obj)
    148         except RuntimeError as e:

/opt/anaconda3/lib/python3.6/pickle.py in dump(self, obj)
    408             self.framer.start_framing()
--> 409         self.save(obj)
    410         self.write(STOP)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_tuple(self, obj)
    735             for element in obj:
--> 736                 save(element)
    737             # Subtle.  Same as in the big comment below.

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_function(self, obj, name)
    247             #print("save global", islambda(obj), obj.__code__.co_filename, modname, themodule)
--> 248             self.save_function_tuple(obj)
    249             return

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_function_tuple(self, func)
    295         # save the rest of the func data needed by _fill_function
--> 296         save(f_globals)
    297         save(defaults)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_dict(self, obj)
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822

/opt/anaconda3/lib/python3.6/pickle.py in _batch_setitems(self, items)
    846                     save(k)
--> 847                     save(v)
    848                 write(SETITEMS)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_function(self, obj, name)
    247             #print("save global", islambda(obj), obj.__code__.co_filename, modname, themodule)
--> 248             self.save_function_tuple(obj)
    249             return

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_function_tuple(self, func)
    295         # save the rest of the func data needed by _fill_function
--> 296         save(f_globals)
    297         save(defaults)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_dict(self, obj)
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822

/opt/anaconda3/lib/python3.6/pickle.py in _batch_setitems(self, items)
    846                     save(k)
--> 847                     save(v)
    848                 write(SETITEMS)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    520         # Save the reduce() output and finally memoize the object
--> 521         self.save_reduce(obj=obj, *rv)
    522

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    565             args = args[1:]
--> 566             save(cls)
    567

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_global(self, obj, name, pack)
    430                 dd[k] = v
--> 431             self.save(dd)
    432             self.write(pickle.TUPLE2)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_dict(self, obj)
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822

/opt/anaconda3/lib/python3.6/pickle.py in _batch_setitems(self, items)
    846                     save(k)
--> 847                     save(v)
    848                 write(SETITEMS)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_function(self, obj, name)
    247             #print("save global", islambda(obj), obj.__code__.co_filename, modname, themodule)
--> 248             self.save_function_tuple(obj)
    249             return

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_function_tuple(self, func)
    295         # save the rest of the func data needed by _fill_function
--> 296         save(f_globals)
    297         save(defaults)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_dict(self, obj)
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822

/opt/anaconda3/lib/python3.6/pickle.py in _batch_setitems(self, items)
    846                     save(k)
--> 847                     save(v)
    848                 write(SETITEMS)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_dict(self, obj)
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822

/opt/anaconda3/lib/python3.6/pickle.py in _batch_setitems(self, items)
    846                     save(k)
--> 847                     save(v)
    848                 write(SETITEMS)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    520         # Save the reduce() output and finally memoize the object
--> 521         self.save_reduce(obj=obj, *rv)
    522

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    580             save(func)
--> 581             save(args)
    582             write(pickle.REDUCE)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_tuple(self, obj)
    750         for element in obj:
--> 751             save(element)
    752

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    520         # Save the reduce() output and finally memoize the object
--> 521         self.save_reduce(obj=obj, *rv)
    522

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    580             save(func)
--> 581             save(args)
    582             write(pickle.REDUCE)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_tuple(self, obj)
    735             for element in obj:
--> 736                 save(element)
    737             # Subtle.  Same as in the big comment below.

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_bytes(self, obj)
    692                 self.save_reduce(codecs.encode,
--> 693                                  (str(obj, 'latin1'), 'latin1'), obj=obj)
    694             return

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj)
    580             save(func)
--> 581             save(args)
    582             write(pickle.REDUCE)

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_tuple(self, obj)
    735             for element in obj:
--> 736                 save(element)
    737             # Subtle.  Same as in the big comment below.

/opt/anaconda3/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    475         if f is not None:
--> 476             f(self, obj) # Call unbound method with explicit self
    477             return

/opt/anaconda3/lib/python3.6/pickle.py in save_str(self, obj)
    713             else:
--> 714                 self.write(BINUNICODE + pack("<I", n) + encoded)
    715         else:

error: 'I' format requires 0 <= number <= 4294967295

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<ipython-input-28-9c42c4c210ac> in <module>
     34     return prob
     35
---> 36 one_row_predict_demo_udf = udf(one_row_predict_demo, DoubleType())
     37 df = demo.repartition(100).withColumn('demo_prob', one_row_predict_demo_udf(col('global_media'),col('mediatypefirst'),col('mediatypesecond'),col('camp'),col('industry'),col('business_tag'),
     38             col('appname_list'),col('packagename_list'),col('brand_list'),col('product_list'),col('pkg_attr'),col('city_idx'),col('manu_idx'),col('fc')))

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/sql/functions.py in udf(f, returnType)
   1870     [Row(slen=5), Row(slen=3)]
   1871     """
-> 1872     return UserDefinedFunction(f, returnType)
   1873
   1874 blacklist = ['map', 'since', 'ignore_unicode_prefix']

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/sql/functions.py in __init__(self, func, returnType, name)
   1828         self.returnType = returnType
   1829         self._broadcast = None
-> 1830         self._judf = self._create_judf(name)
   1831
   1832     def _create_judf(self, name):

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/sql/functions.py in _create_judf(self, name)
   1833         from pyspark.sql import SparkSession
   1834         sc = SparkContext.getOrCreate()
-> 1835         wrapped_func = _wrap_function(sc, self.func, self.returnType)
   1836         spark = SparkSession.builder.getOrCreate()
   1837         jdt = spark._jsparkSession.parseDataType(self.returnType.json())

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/sql/functions.py in _wrap_function(sc, func, returnType)
   1813 def _wrap_function(sc, func, returnType):
   1814     command = (func, returnType)
-> 1815     pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command)
   1816     return sc._jvm.PythonFunction(bytearray(pickled_command), env, includes, sc.pythonExec,
   1817                                   sc.pythonVer, broadcast_vars, sc._javaAccumulator)

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/rdd.py in _prepare_for_python_RDD(sc, command)
   2364     # the serialized command will be compressed by broadcast
   2365     ser = CloudPickleSerializer()
-> 2366     pickled_command = ser.dumps(command)
   2367     if len(pickled_command) > (1 << 20):  # 1M
   2368         # The broadcast will have same life cycle as created PythonRDD

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py in dumps(self, obj)
    462
    463     def dumps(self, obj):
--> 464         return cloudpickle.dumps(obj, 2)
    465
    466

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in dumps(obj, protocol)
    701
    702     cp = CloudPickler(file,protocol)
--> 703     cp.dump(obj)
    704
    705     return file.getvalue()

/usr/lib/spark-2.1.3-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/cloudpickle.py in dump(self, obj)
    153             raise
    154         except Exception as e:
--> 155             if "'i' format requires" in e.message:
    156                 msg = "Object too large to serialize: " + e.message
    157             else:

AttributeError: 'error' object has no attribute 'message'

Thanks for the help!


title: Loss per epoch equation: Pytorch

body:
Im new to Pytorch and Im confused if my loss per epoch equation is correct or not because my val_loss is lower than by train_loss. The code is as follows.
from torch import optim
def fit(train_dl,model,test_dl=None,epoch=1000,lr=0.001):

  criterion = nn.CrossEntropyLoss()
  optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

  for epoch in range(epoch):
      running_loss = 0.0
      model.train()
      for i,(inputs, labels) in enumerate(train_dl):

          optimizer.zero_grad()

          outputs = model(inputs)
          loss = criterion(outputs, labels)

          loss.backward()
          optimizer.step()
          running_loss += loss.item()*inputs.size(0)

          # if i % 2000 == 1999:
          #     print('[%d, %5d] loss: %.3f' %
          #           (epoch + 1, i + 1, running_loss/2000))
          #     running_loss = 0.0
      print("Epoch: ",epoch+1)
      print("Train Loss: ",running_loss/len(train_dl.sampler))

      if test_dl is not None:
          model.eval()
          running_loss = 0
          for i,(inputs, labels) in enumerate(test_dl):
              outputs = model(inputs)
              loss = criterion(outputs, labels)
              running_loss += loss.item()*inputs.size(0)
          print("Validation Loss: ",running_loss/len(test_dl.sampler))
      print("\n"*2)

  print('Finished Training')

The loss per epoch of train and val data is as follows.
Epoch:  1

Train Loss:  1.1405704270362853

Validation Loss:  0.6346351503372193

Epoch:  2

Train Loss:  1.109910039997101

Validation Loss:  0.6700555039405822

Epoch:  3

Train Loss:  1.0855869434356689

Validation Loss:  0.5310997007369995

Epoch:  4

Train Loss:  1.0623883997917176

Validation Loss:  0.5400820639610291

Epoch:  5

Train Loss:  1.040262179851532

Validation Loss:  0.596698025226593

Epoch:  6

Train Loss:  1.0253396962165833

Validation Loss:  0.5367399935722351

Epoch:  7

Train Loss:  1.0021697312355042

Validation Loss:  0.48242439060211184

Epoch:  8

Train Loss:  0.9849149712562562

Validation Loss:  0.46645955047607424

Epoch:  9

Train Loss:  0.9655076077461243

Validation Loss:  0.5602270641326904

Epoch:  10

Train Loss:  0.9523622084617615

Validation Loss:  0.49749606523513795

Epoch:  11

Train Loss:  0.9366362886428833

Validation Loss:  0.4111686680316925

Epoch:  12

Train Loss:  0.9228082064628601

Validation Loss:  0.47805821266174314

Epoch:  13

Train Loss:  0.9069208443641663

Validation Loss:  0.3824161283493042

Epoch:  14

Train Loss:  0.8945751609802246

Validation Loss:  0.5301653007507324

Epoch:  15

Train Loss:  0.8807038812637329

Validation Loss:  0.41203391284942625

Epoch:  16

Train Loss:  0.8668060901641845

Validation Loss:  0.4474089996814728

Epoch:  17

Train Loss:  0.853671452999115

Validation Loss:  0.5498352452278137

Epoch:  18

Train Loss:  0.8409735582351685

Validation Loss:  0.3281970458984375

Epoch:  19

Train Loss:  0.8269806921958923

Validation Loss:  0.3685186356544495

Epoch:  20

Train Loss:  0.8118648974895477

Validation Loss:  0.4514490584373474

Finished Training


title: PyTorch Tensorboard not as described in documentation

body:
I'm using PyTorch's ùtils.tensorboard.writer to log training of an RNN. For the àdd_hparams() function the Docs say:

Params:

hparam_dict (dict) – Each key-value pair in the dictionary is the name of the hyper parameter and it’s corresponding value. The type of the value can be one of bool, string, float, int, or None.
metric_dict (dict) – Each key-value pair in the dictionary is the name of the metric and it’s corresponding value. Note that the key used here should be unique in the tensorboard record. Otherwise the value you added by add_scalar will be displayed in hparam plugin. In most cases, this is unwanted.
hparam_domain_discrete – (Optional[Dict[str, List[Any]]]) A dictionary that contains names of the hyperparameters and all discrete values they can hold
run_name (str) – Name of the run, to be included as part of the logdir. If unspecified, will use current timestamp.

Source: https://pytorch.org/docs/master/tensorboard.html

But when I try to use the run_name parameter, I get the error

TypeError: add_hparams() got an unexpected keyword argument 'run_name'

So I looked up the writer.py file I imported and found the cause to be the add_hparams() function itself:
def add_hparams(self, hparam_dict, metric_dict):

I checked my installation of PyTorch, but it's up to date. Is this some sort of nightly feature, if so, how can I download the nightly version of torch?


title: PyTorch Same Input Different Output (Not Random)

body:
I have a model that I have trained, and I am testing it by running it (on .eval() mode).
Here are the exact lines and order that I've executed in debugger:
(Pdb) p feature
tensor([[[ -4.0563,  -3.8415,  -3.9542,  ..., -14.8424, -14.9201, -14.8960],
         [ -5.8481,  -2.0405,  -2.4438,  ..., -19.6938, -19.4901, -19.9180],
         [ -5.2424,  -1.2804,  -1.5109,  ..., -19.3892, -19.4397, -19.5012],
         ...,
         [ -6.4756,  -2.0376,  -2.0894,  ..., -20.0942, -19.9635, -19.8762],
         [ -6.5087,  -2.0452,  -1.9018,  ..., -19.7127, -19.8574, -20.0103],
         [ -7.0725,  -4.2817,  -3.3231,  ..., -16.7170, -16.9004, -17.0333]]])
(Pdb) p feature2
tensor([[[ -4.0563,  -3.8415,  -3.9542,  ..., -14.8424, -14.9201, -14.8960],
         [ -5.8481,  -2.0405,  -2.4438,  ..., -19.6938, -19.4901, -19.9180],
         [ -5.2424,  -1.2804,  -1.5109,  ..., -19.3892, -19.4397, -19.5012],
         ...,
         [ -6.4756,  -2.0376,  -2.0894,  ..., -20.0942, -19.9635, -19.8762],
         [ -6.5087,  -2.0452,  -1.9018,  ..., -19.7127, -19.8574, -20.0103],
         [ -7.0725,  -4.2817,  -3.3231,  ..., -16.7170, -16.9004, -17.0333]]])
(Pdb) torch.all(feature == feature2)
tensor(True)
(Pdb) prediction_tag, prediction_time = model(feature)
(Pdb) prediction_tag2, prediction_time2 = model(feature2)
(Pdb) prediction_time
tensor([[[9.6584e-06, 3.9059e-05, 4.0984e-06,  ..., 1.7644e-04,
          1.0589e-02, 4.4167e-06],
         [9.6584e-06, 3.9059e-05, 4.0984e-06,  ..., 1.7644e-04,
          1.0589e-02, 4.4167e-06],
         [9.3993e-06, 3.7754e-05, 3.9786e-06,  ..., 1.7362e-04,
          1.0243e-02, 4.2382e-06],
         ...,
         [7.8885e-06, 1.1077e-05, 3.8594e-06,  ..., 1.9443e-04,
          3.8032e-03, 6.6878e-06],
         [8.0696e-06, 1.1217e-05, 3.9580e-06,  ..., 2.0004e-04,
          3.7598e-03, 6.8072e-06],
         [8.0696e-06, 1.1217e-05, 3.9580e-06,  ..., 2.0004e-04,
          3.7598e-03, 6.8072e-06]]])
(Pdb) p prediction_time2
tensor([[[8.0289e-07, 2.0557e-05, 2.5803e-05,  ..., 3.3225e-04,
          4.4547e-03, 8.4192e-06],
         [8.0289e-07, 2.0557e-05, 2.5803e-05,  ..., 3.3225e-04,
          4.4547e-03, 8.4192e-06],
         [7.6509e-07, 1.9805e-05, 2.4918e-05,  ..., 3.2385e-04,
          4.3618e-03, 7.9963e-06],
         ...,
         [7.3927e-07, 8.7688e-06, 1.8454e-05,  ..., 1.9831e-04,
          1.9305e-03, 6.2879e-06],
         [7.7376e-07, 8.8673e-06, 1.8517e-05,  ..., 2.0194e-04,
          1.8297e-03, 6.3183e-06],
         [7.7376e-07, 8.8673e-06, 1.8517e-05,  ..., 2.0194e-04,
          1.8297e-03, 6.3183e-06]]])
(Pdb) torch.all(prediction_time == prediction_time2)
tensor(False)

As you can see, even though feature and feature2 are seemingly the same inputs, the outputs by the model do not match. This isn't random either, as after I have executed those lines above and running these lines below:
(Pdb) prediction_tag, prediction_time = model(feature)
(Pdb) prediction_time
tensor([[[9.6584e-06, 3.9059e-05, 4.0984e-06,  ..., 1.7644e-04,
          1.0589e-02, 4.4167e-06],
         [9.6584e-06, 3.9059e-05, 4.0984e-06,  ..., 1.7644e-04,
          1.0589e-02, 4.4167e-06],
         [9.3993e-06, 3.7754e-05, 3.9786e-06,  ..., 1.7362e-04,
          1.0243e-02, 4.2382e-06],
         ...,
         [7.8885e-06, 1.1077e-05, 3.8594e-06,  ..., 1.9443e-04,
          3.8032e-03, 6.6878e-06],
         [8.0696e-06, 1.1217e-05, 3.9580e-06,  ..., 2.0004e-04,
          3.7598e-03, 6.8072e-06],
         [8.0696e-06, 1.1217e-05, 3.9580e-06,  ..., 2.0004e-04,
          3.7598e-03, 6.8072e-06]]])
(Pdb) prediction_tag2, prediction_time2 = model(feature2)
(Pdb) prediction_time2
tensor([[[8.0289e-07, 2.0557e-05, 2.5803e-05,  ..., 3.3225e-04,
          4.4547e-03, 8.4192e-06],
         [8.0289e-07, 2.0557e-05, 2.5803e-05,  ..., 3.3225e-04,
          4.4547e-03, 8.4192e-06],
         [7.6509e-07, 1.9805e-05, 2.4918e-05,  ..., 3.2385e-04,
          4.3618e-03, 7.9963e-06],
         ...,
         [7.3927e-07, 8.7688e-06, 1.8454e-05,  ..., 1.9831e-04,
          1.9305e-03, 6.2879e-06],
         [7.7376e-07, 8.8673e-06, 1.8517e-05,  ..., 2.0194e-04,
          1.8297e-03, 6.3183e-06],
         [7.7376e-07, 8.8673e-06, 1.8517e-05,  ..., 2.0194e-04,
          1.8297e-03, 6.3183e-06]]])

I get the same, different outputs. Why am I experiencing this issue? I am totally confused.
Notes: I have checked that both feature and feature2 have dtypes of torch.float32. feature was extracted from a torch DataLoader that was set up, while feature2 was directly obtained from reading a file.
EDIT: Here is how the model was constructed:
class CRNN(nn.Module):
    def __init__(self, inputdim, outputdim, **kwargs):
        super().__init__()
        features = nn.ModuleList()
        self.features = nn.Sequential(
            Block2D(1, 32),
            nn.LPPool2d(4, (2, 4)),
            Block2D(32, 128),
            Block2D(128, 128),
            nn.LPPool2d(4, (2, 4)),
            Block2D(128, 128),
            Block2D(128, 128),
            nn.LPPool2d(4, (1, 4)),
            nn.Dropout(0.3),
        )
        with torch.no_grad():
            rnn_input_dim = self.features(torch.randn(1, 1, 500,
                                                      inputdim)).shape
            rnn_input_dim = rnn_input_dim[1] * rnn_input_dim[-1]

        self.gru = nn.GRU(rnn_input_dim,
                          128,
                          bidirectional=True,
                          batch_first=True)
        self.temp_pool = parse_poolingfunction(kwargs.get(
            'temppool', 'linear'),
                                               inputdim=256,
                                               outputdim=outputdim)
        self.outputlayer = nn.Linear(256, outputdim)
        self.features.apply(init_weights)
        self.outputlayer.apply(init_weights)

    def forward(self, x):
        batch, time, dim = x.shape
        x = x.unsqueeze(1)
        x = self.features(x)
        x = x.transpose(1, 2).contiguous().flatten(-2)
        x, _ = self.gru(x)
        decision_time = torch.sigmoid(self.outputlayer(x)).clamp(1e-7, 1.)
        decision_time = torch.nn.functional.interpolate(
            decision_time.transpose(1, 2),
            time,
            mode='linear',
            align_corners=False).transpose(1, 2)
        decision = self.temp_pool(x, decision_time).clamp(1e-7, 1.).squeeze(1)
        return decision, decision_time

def crnn(inputdim=64, outputdim=527, pretrained_file='gpv_f'):
    model = CRNN(inputdim, outputdim)
    if pretrained_file:
        state = torch.load(Path(__file__).parent / pretrained_file,
                           map_location='cpu')
        model.load_state_dict(state, strict=True)
    return model

with the following helpers:
class Block2D(nn.Module):
    def __init__(self, cin, cout, kernel_size=3, padding=1):
        super().__init__()
        self.block = nn.Sequential(
            nn.BatchNorm2d(cin),
            nn.Conv2d(cin,
                      cout,
                      kernel_size=kernel_size,
                      padding=padding,
                      bias=False),
            nn.LeakyReLU(inplace=True, negative_slope=0.1))

    def forward(self, x):
        return self.block(x)

def init_weights(m):
    if isinstance(m, (nn.Conv2d, nn.Conv1d)):
        nn.init.kaiming_normal_(m.weight)
        if m.bias is not None:
            nn.init.constant_(m.bias, 0)
    elif isinstance(m, nn.BatchNorm2d):
        nn.init.constant_(m.weight, 1)
        if m.bias is not None:
            nn.init.constant_(m.bias, 0)
    if isinstance(m, nn.Linear):
        nn.init.kaiming_uniform_(m.weight)
        if m.bias is not None:
            nn.init.constant_(m.bias, 0)

class LinearSoftPool(nn.Module):
    """LinearSoftPool

    Linear softmax, takes logits and returns a probability, near to the actual maximum value.
    Taken from the paper:

        A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling
    https://arxiv.org/abs/1810.09050

    """
    def __init__(self, pooldim=1):
        super().__init__()
        self.pooldim = pooldim

    def forward(self, logits, time_decision):
        return (time_decision**2).sum(self.pooldim) / time_decision.sum(
            self.pooldim)

class MeanPool(nn.Module):
    def __init__(self, pooldim=1):
        super().__init__()
        self.pooldim = pooldim

    def forward(self, logits, decision):
        return torch.mean(decision, dim=self.pooldim)

def parse_poolingfunction(poolingfunction_name='mean', **kwargs):
    """parse_poolingfunction
    A heler function to parse any temporal pooling
    Pooling is done on dimension 1

    :param poolingfunction_name:
    :param **kwargs:
    """
    poolingfunction_name = poolingfunction_name.lower()
    if poolingfunction_name == 'mean':
        return MeanPool(pooldim=1)
    elif poolingfunction_name == 'linear':
        return LinearSoftPool(pooldim=1)


title: How can one utilize the indices provided by torch.topk()?

body:
Suppose I have a pytorch tensor x of shape [N, N_g, 2]. It can be viewed as N * N_g 2d vectors. Specifically, x[i, j, :] is the 2d vector of the jth group in the ith batch.
Now I am trying to get the coordinates of vectors of top 5 length in each group. So I tried the following:
(i) First I used x_len = (x**2).sum(dim=2).sqrt() to compute their lengths, resulting in x_len.shape==[N, N_g].
(ii) Then I used tk = x_len.topk(5) to get the top 5 lengths in each group.
(iii) The desired output would be a tensor x_top5 of shape [N, 5, 2]. Naturally I thought of using tk.indices to index x so as to obtain x_top5. But I failed as it seems such indexing is not supported.
How can I do this?

A minimal example:
x = torch.randn(10,10,2) # N=10 is the batchsize, N_g=10 is the group size
x_len = (x**2).sum(dim=2).sqrt()
tk = x_len.topk(5)

x_top5 = x[tk.indices]

print(x_top5.shape)
# torch.Size([10, 5, 10, 2])

However, this gives x_top5 as a tensor of shape [10, 5, 10, 2], instead of [10, 5, 2] as desired.


title: How to add a new dim to a a pytorch tensor?

body:
In numpy I would do
a = np.zeros((4, 5, 6))
a = a[:, :, np.newaxis, :]
assert a.shape == (4, 5, 1, 6)

How to do the same in torch?


title: Torch throws a RuntimeError: element 0 of tensors does not require grad... but can't find where computational graph is severed

body:
I am getting the above error:
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
I looked this up and it looks like the computational graph is not connected for some reason.  However, I cannot find the location where the graph is severed.
My code is a reproduction of the arjovsky WGAN: https://github.com/martinarjovsky/WassersteinGAN
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import torch
import torch.nn as nn

from __future__ import print_function
import random
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
from torch.autograd import Variable
import os
import json

class MLP_G(nn.Module):
    def __init__(self, isize, nz, ngf, ngpu):
        super(MLP_G, self).__init__()
        self.ngpu = ngpu

        main = nn.Sequential(
            # Z goes into a linear of size: ngf
            nn.Linear(nz, ngf),
            nn.ReLU(True),
            nn.Linear(ngf, ngf),
            nn.ReLU(True),
            nn.Linear(ngf, ngf),
            nn.ReLU(True),
            nn.Linear(ngf, isize),
        )
        self.main = main
        self.isize = isize
        self.nz = nz

    def forward(self, input):
        input = input.view(input.size(0), input.size(1))
        if isinstance(input.data, torch.cuda.FloatTensor) and self.ngpu > 1:
            output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))
        else:
            output = self.main(input)
        return output.view(output.size(0), self.isize)
class MLP_D(nn.Module):
    def __init__(self, isize, nz, ndf, ngpu):
        super(MLP_D, self).__init__()
        self.ngpu = ngpu

        main = nn.Sequential(
            # Z goes into a linear of size: ndf
            nn.Linear(isize, ndf),
            nn.ReLU(True),
            nn.Linear(ndf, ndf),
            nn.ReLU(True),
            nn.Linear(ndf, ndf),
            nn.ReLU(True),
            nn.Linear(ndf, 1),
        )
        self.main = main
        self.isize = isize
        self.nz = nz

    def forward(self, input):
        input = input.view(input.size(0),input.size(1))
        if isinstance(input.data, torch.cuda.FloatTensor) and self.ngpu > 1:
            output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))
        else:
            output = self.main(input)
        output = output.mean(0)
        return output.view(1)

netG = None #path to saved generator
netD = None #discriminator path
batchSize = 1000 #size of batch (which is size of data)
cuda = False
lrD = lrG = .00005
beta1 = .5
niter = 25
experiment = '/content/drive/MyDrive/savefolder'
clamp_upper = .01
clamp_lower = -clamp_upper

manualSeed = random.randint(1, 10000) # fix seed
print("Random Seed: ", manualSeed)
random.seed(manualSeed)
torch.manual_seed(manualSeed)

cudnn.benchmark = True

dataset = torch.tensor(np.stack([x,y, instrument], axis = 1)).float().reshape(-1,3)

ngpu = 1
nz = 4 #three latents and the instrument
ngf = 128
ndf = 128

# custom weights initialization called on netG and netD
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

netG = MLP_G(2, nz, ngf, ngpu)

netG.apply(weights_init)

print(netG)

netD = MLP_D(3, nz, ndf, ngpu)

print(netD)

input = torch.FloatTensor(batchSize, 2)
noise = torch.FloatTensor(batchSize, nz-1)
fixed_noise = torch.FloatTensor(batchSize, nz-1).normal_(0, 1)
one = torch.FloatTensor([1])
mone = one * -1

# setup optimizer

optimizerD = optim.Adam(netD.parameters(), lr=lrD, betas=(beta1, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr=lrG, betas=(beta1, 0.999))

real_cpu = data = dataset

gen_iterations = 0
for epoch in range(niter):
    #data_iter = iter(dataloader)
    ############################
    # (1) Update D network
    ###########################
    for p in netD.parameters(): # reset requires_grad
        p.requires_grad = True # they are set to False below in netG update

    # train the discriminator Diters times
    if gen_iterations < 25 or gen_iterations % 500 == 0:
        Diters = 100
    else:
        Diters = 5
    j = 0
    while j < Diters:
        j += 1

        # clamp parameters to a cube
        for p in netD.parameters():
            p.data.clamp_(clamp_lower, clamp_upper)

        # train with real
        netD.zero_grad()

        if cuda:
            real_cpu = real_cpu.cuda()
        input.resize_as_(real_cpu).copy_(real_cpu)
        inputv = Variable(input, requires_grad=False)

        errD_real = netD(inputv)
        errD_real.backward(one)#Error Occurs here

        # train with fake
        noise.resize_(batchSize, nz-1).normal_(0, 1)
        noisev = torch.cat([Variable(noise, requires_grad=False), dataset[:,2].reshape(-1,1)], 1)# totally freeze netG
        fake = torch.cat([Variable(netG(noisev).data), dataset[:,2].view(-1,1)], 1)
        inputv = fake
        errD_fake = netD(inputv)
        errD_fake.backward(mone)
        errD = errD_real - errD_fake
        optimizerD.step()

        ############################
        # (2) Update G network
        ###########################
        for p in netD.parameters():
            p.requires_grad = False # to avoid computation
        netG.zero_grad()
        # in case our last batch was the tail batch of the dataloader,
        # make sure we feed a full batch of noise
        noise.resize_(batchSize, nz-1).normal_(0, 1)
        noisev = torch.cat([Variable(noise), dataset[:,2].view(-1,1)], 1)
        fake = torch.cat([netG(noisev), dataset[:,2].view(-1,1)], 1)
        errG = netD(fake)
        errG.backward(one)
        optimizerG.step()
        gen_iterations += 1

        i = 0

        print('[%d/%d][%d] Loss_D: %f Loss_G: %f Loss_D_real: %f Loss_D_fake %f'
            % (epoch, niter, gen_iterations,
            errD.data[0], errG.data[0], errD_real.data[0], errD_fake.data[0]))
        # if gen_iterations % 500 == 0:
        #     real_cpu = real_cpu.mul(0.5).add(0.5)
        #     vutils.save_image(real_cpu, '{0}/real_samples.png'.format(opt.experiment))
        #     fake = netG(Variable(fixed_noise, volatile=True))
        #     fake.data = fake.data.mul(0.5).add(0.5)
        #     vutils.save_image(fake.data, '{0}/fake_samples_{1}.png'.format(opt.experiment, gen_iterations))

    # do checkpointing
    torch.save(netG.state_dict(), '{0}/netG_epoch_{1}.pth'.format(experiment, epoch))
    torch.save(netD.state_dict(), '{0}/netD_epoch_{1}.pth'.format(experiment, epoch))

Error occurs on the line: errD_real.backward(one).  The error might be something regarding zeroing out the computational graph as the code runs for one iteration then throws an error.  Thanks for your help.


title: pytorch: zero_grad vs zero_grad() - role of parenthesis?

body:
I'm trying to train a pytorch model as follows:

start = time.time()

for epoch in range(100):

    t_loss = 0
    for i in range(100):

        optimizer.zero_grad
        scores = my_model(sent_dict_list[i])
        scores = scores.permute(0, 2, 1)

        loss = loss_function(scores, torch.tensor(targ_list[i]).cuda())
        t_loss += loss.item()
        loss.backward()
        optimizer.step()

    print("t_loss = ", t_loss)

I find that when I call "optimizer.zero_grad" my loss decreases at the end of every epoch whereas when I call "optimizer.zero_grad()" with the parenthesis it stays almost exactly the same. I don't know what difference this makes and was hoping someone could explain it to me.


title: Python package installation issues: discrepancy between installation check on terminal (Anaconda Prompt) and Jupyter Notebook

body:
I have to install the Paysage package. I downloaded the GitHub repo and moved in it to launch the installation command.

Beforehand, I had Python 3.5.2 installed on my system (Windows10) with some packages such as NumPy, SciPy, Pandas, Matplotlib and others, and Jupyter notebook. The package documentation recommends to set up Paysage within an Anaconda environment, so I thought neat to install Anaconda3. This latter installed on my system Python 3.7.2. So now I have Python 3.7.2 under conda and Python 3.5.2 if for example I check the version with the cmd command python --version.

I think I have been too loose about it because now I could set up Paysage (apparently) properly, by having created an environment, moved to the Paysage folder where the setup.py lays, and did

pip install .

After a trial-error route, to gather all what needed in order to install the package, I finally succeeded. In fact, as I check whether Paysage is available, by running the script

try:
    import paysage
except ImportError:
    available = False
    print('Paysage not available')
else:
    available = True
    print('Paysage available')

it turns out that Paysage is indeed installed. As the script is executed, a message confirms it is ok:

(paysageTest) C:\Users\...>python test.py
Running paysage with the pytorch backend on the cpu

But the problem rises in the moment in which I try to do the same in a Jupyter Notebook. I have some guidelines and examples on notebooks and I would like to use them. If I type the same code as above in a Jupyter Notebook cell, it turns out that Paysage is unknown to the kernel running. So it seems that even if Paysage is installed, it can not be run by the notebook.

It is maybe due to the discrepancy between the Python installed version on the system and the one installed within the Anaconda? (The Jupyter notebook has been launched from an Anaconda Prompt, created for the Paysage setup purpose, as in the docs). It is maybe due to the fact that launching a notebook, even if from the Anaconda Prompt, that notebook relies on the Python version installed on the computer and thus version incompatibilities are lurking?