I am trying to setup caffe2 without gpu in windows and I am getting the following error:

I have tried installation from caffe2 page but that didn't work. So I have tried the steps from the pytorch's github :
https://github.com/pytorch/pytorch#binaries

I have ran this from VSCODe by Anaconda.
I have instlled all the dependencies.

I am building from source. I ran build_winows.bat from vscode.

[1996/2139] Building CXX object caffe2\torc...hon.dir\csrc\jit\python_interpreter.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[1997/2139] Building CXX object caffe2\torc...torch_python.dir\csrc\jit\python_ir.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[1998/2139] Building CXX object caffe2\torc...iles\torch_python.dir\csrc\jit\init.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2000/2139] Building CXX object caffe2\torc...ython.dir\csrc\multiprocessing\init.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2001/2139] Building CXX object caffe2\torc...hon.dir\csrc\jit\python_arg_flatten.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2002/2139] Building CXX object caffe2\torc...src\jit\script\python_sugared_value.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
..\torch\csrc\jit\script\python_sugared_value.cpp(30): warning C4273: 'torch::jit::script::getRecursiveScriptMode': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/script/python_sugared_value.h(195): note: see previous definition of 'getRecursiveScriptMode'
[2004/2139] Building CXX object caffe2\torc...rch_python.dir\csrc\jit\script\init.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2005/2139] Building CXX object caffe2\torc...keFiles\torch_python.dir\csrc\utils.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2006/2139] Building CXX object caffe2\torc...dir\csrc\utils\throughput_benchmark.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2007/2139] Building CXX object caffe2\torc...es\torch_python.dir\csrc\utils\init.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2008/2139] Building CXX object caffe2\torc...les\torch_python.dir\csrc\onnx\init.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2010/2139] Building CXX object caffe2\torc...torch_python.dir\csrc\serialization.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2012/2139] Building CXX object caffe2\torc...ython.dir\csrc\tensor\python_tensor.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2013/2139] Building CXX object caffe2\torc..._python.dir\csrc\utils\tensor_apply.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2016/2139] Building CXX object caffe2\torc...on.dir\csrc\utils\python_arg_parser.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2021/2139] Building CXX object caffe2\torc...h_python.dir\csrc\utils\tensor_list.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2023/2139] Building CXX object caffe2\torc...ch_python.dir\csrc\utils\tensor_new.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2025/2139] Building CXX object caffe2\torc..._python.dir\csrc\utils\tuple_parser.cpp.obj
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(302): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(290): note: see previous definition of 'addInputs'
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(309): warning C4273: 'torch::jit::tracer::addInputs': inconsistent dll linkage
D:\Caffe2 Project\pytorch\torch/csrc/jit/tracer.h(296): note: see previous definition of 'addInputs'
[2026/2139] Building CXX object modules\det...fe2_detectron_ops.dir\ps_roi_pool_op.cc.obj
..\modules\detectron\ps_roi_pool_op.cc(26): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_PSRoIPool': inconsistent dll linkage
..\modules\detectron\ps_roi_pool_op.cc(21): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_PSRoIPool'
..\modules\detectron\ps_roi_pool_op.cc(69): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_PSRoIPoolGradient': inconsistent dll linkage
..\modules\detectron\ps_roi_pool_op.cc(22): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_PSRoIPoolGradient'
[2027/2139] Building CXX object modules\det..._detectron_ops.dir\smooth_l1_loss_op.cc.obj
..\modules\detectron\smooth_l1_loss_op.cc(26): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SmoothL1Loss': inconsistent dll linkage
..\modules\detectron\smooth_l1_loss_op.cc(21): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SmoothL1Loss'
..\modules\detectron\smooth_l1_loss_op.cc(76): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SmoothL1LossGradient': inconsistent dll linkage
..\modules\detectron\smooth_l1_loss_op.cc(22): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SmoothL1LossGradient'
[2028/2139] Building CXX object modules\det...ffe2_detectron_ops.dir\roi_pool_f_op.cc.obj
..\modules\detectron\roi_pool_f_op.cc(24): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_RoIPoolF': inconsistent dll linkage
..\modules\detectron\roi_pool_f_op.cc(21): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_RoIPoolF'
..\modules\detectron\roi_pool_f_op.cc(62): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_RoIPoolFGradient': inconsistent dll linkage
..\modules\detectron\roi_pool_f_op.cc(22): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_RoIPoolFGradient'
[2029/2139] Building CXX object modules\det...ron_ops.dir\select_smooth_l1_loss_op.cc.obj
..\modules\detectron\select_smooth_l1_loss_op.cc(28): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SelectSmoothL1Loss': inconsistent dll linkage
..\modules\detectron\select_smooth_l1_loss_op.cc(21): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SelectSmoothL1Loss'
..\modules\detectron\select_smooth_l1_loss_op.cc(66): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SelectSmoothL1LossGradient': inconsistent dll linkage
..\modules\detectron\select_smooth_l1_loss_op.cc(24): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SelectSmoothL1LossGradient'
[2030/2139] Building CXX object modules\det...ectron_ops.dir\sigmoid_focal_loss_op.cc.obj
..\modules\detectron\sigmoid_focal_loss_op.cc(26): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SigmoidFocalLoss': inconsistent dll linkage
..\modules\detectron\sigmoid_focal_loss_op.cc(21): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SigmoidFocalLoss'
..\modules\detectron\sigmoid_focal_loss_op.cc(80): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SigmoidFocalLossGradient': inconsistent dll linkage
..\modules\detectron\sigmoid_focal_loss_op.cc(22): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SigmoidFocalLossGradient'
[2031/2139] Building CXX object modules\det...ps.dir\sigmoid_cross_entropy_loss_op.cc.obj
..\modules\detectron\sigmoid_cross_entropy_loss_op.cc(28): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SigmoidCrossEntropyLoss': inconsistent dll linkage
..\modules\detectron\sigmoid_cross_entropy_loss_op.cc(21): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SigmoidCrossEntropyLoss'
..\modules\detectron\sigmoid_cross_entropy_loss_op.cc(63): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SigmoidCrossEntropyLossGradient': inconsistent dll linkage
..\modules\detectron\sigmoid_cross_entropy_loss_op.cc(24): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SigmoidCrossEntropyLossGradient'
[2032/2139] Building CXX object modules\det...affe2_detectron_ops.dir\sample_as_op.cc.obj
..\modules\detectron\sample_as_op.cc(24): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SampleAs': inconsistent dll linkage
..\modules\detectron\sample_as_op.cc(21): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SampleAs'
..\modules\detectron\sample_as_op.cc(47): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SampleAsGradient': inconsistent dll linkage
..\modules\detectron\sample_as_op.cc(22): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SampleAsGradient'
[2033/2139] Building CXX object modules\det...tectron_ops.dir\spatial_narrow_as_op.cc.obj
..\modules\detectron\spatial_narrow_as_op.cc(26): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SpatialNarrowAs': inconsistent dll linkage
..\modules\detectron\spatial_narrow_as_op.cc(21): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SpatialNarrowAs'
..\modules\detectron\spatial_narrow_as_op.cc(48): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SpatialNarrowAsGradient': inconsistent dll linkage
..\modules\detectron\spatial_narrow_as_op.cc(22): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SpatialNarrowAsGradient'
[2034/2139] Building CXX object modules\det...ectron_ops.dir\softmax_focal_loss_op.cc.obj
..\modules\detectron\softmax_focal_loss_op.cc(28): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SoftmaxFocalLoss': inconsistent dll linkage
..\modules\detectron\softmax_focal_loss_op.cc(23): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SoftmaxFocalLoss'
..\modules\detectron\softmax_focal_loss_op.cc(78): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SoftmaxFocalLossGradient': inconsistent dll linkage
..\modules\detectron\softmax_focal_loss_op.cc(24): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_SoftmaxFocalLossGradient'
[2035/2139] Building CXX object modules\mod...test_dynamic.dir\module_test_dynamic.cc.obj
..\modules\module_test\module_test_dynamic.cc(39): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_Caffe2ModuleTestDynamicDummy': inconsistent dll linkage
..\modules\module_test\module_test_dynamic.cc(37): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_Caffe2ModuleTestDynamicDummy'
[2036/2139] Building CXX object modules\det...etectron_ops.dir\upsample_nearest_op.cc.obj
..\modules\detectron\upsample_nearest_op.cc(35): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_UpsampleNearest': inconsistent dll linkage
..\modules\detectron\upsample_nearest_op.cc(30): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_UpsampleNearest'
..\modules\detectron\upsample_nearest_op.cc(54): warning C4273: 'caffe2::CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_UpsampleNearestGradient': inconsistent dll linkage
..\modules\detectron\upsample_nearest_op.cc(31): note: see previous definition of 'CAFFE2_PLEASE_ADD_OPERATOR_SCHEMA_FOR_UpsampleNearestGradient'
[2037/2139] Linking CXX shared library bin\torch.dll
FAILED: bin/torch.dll lib/torch.lib
cmd.exe /C "cd . && D:\Anaconda\Library\bin\cmake.exe -E vs_link_dll --intdir=caffe2\CMakeFiles\torch.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100177~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100177~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~4\2019\COMMUN~1\VC\Tools\MSVC\1421~1.277\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\torch.rsp  /out:bin\torch.dll /implib:lib\torch.lib /pdb:bin\torch.pdb /dll /version:0.0 /INCREMENTAL:NO   && cd
."
LINK: command "C:\PROGRA~2\MICROS~4\2019\COMMUN~1\VC\Tools\MSVC\1421~1.277\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\torch.rsp /out:bin\torch.dll /implib:lib\torch.lib /pdb:bin\torch.pdb /dll /version:0.0 /INCREMENTAL:NO /MANIFEST /MANIFESTFILE:bin\torch.dll.manifest" failed (exit code 1181) with the following output:
LINK : fatal error LNK1181: cannot open input file 'Project\pytorch\build\lib\caffe2_protos.lib'
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 753, in <module>
    build_deps()
  File "setup.py", line 327, in build_deps
    cmake=cmake)
  File "D:\Caffe2 Project\pytorch\tools\build_pytorch_libs.py", line 64, in build_caffe2
    cmake.build(my_env)
  File "D:\Caffe2 Project\pytorch\tools\setup_helpers\cmake.py", line 354, in build
    self.run(build_args, my_env)
  File "D:\Caffe2 Project\pytorch\tools\setup_helpers\cmake.py", line 110, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "D:\Anaconda\lib\subprocess.py", line 341, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '4']' returned non-zero exit status 1.


title: How to use BERT just for ENTITY extraction from a Sequence without classification in the NER task?

body:
My requirement here is given a sentence(sequence), I would like to just extract the entities present in the sequence without classifying them to a type in the NER task. I see that BertForTokenClassification for NER does the classification. Can this be adapted for just the extraction?

Can BERT just be used to do entity extraction/identification?


title: how can I install PyTorch?

body:
I Install PyTorch in Anaconda But when I write :
pip install torchvision
In anaconda this Error show me :
No matching distribution found for torch==1.4.0 (from torchvision)
Did I install it badly?


title: (NLP, TextCNN) Why in_channels equals embedding size in torch.nn.Conv1d?

body:
If we apply torch.nn.Conv2d to a "RGB" image which can be understood as 3 two-dimensional matrices, so parameter in_channels corresponds to the 3 channels 'R', 'G' and 'B'. And in my view, an embedded sentence whose shape is [sentence length, embedding size] should be considered as 1 two-dimensional matrix, so in this case, why parameter in_channels is not 1 but embedding size in torch.nn.Conv1d, not the same meaning as torch.nn.Conv2d?
Could you explain what's the true meaning of in_channels in torch.nn.Conv1d in nlp / TextCNN? Why it's different from torch.nn.Conv2d?
Thanks!


title: Should a data batch be moved to CPU and converted (from torch Tensor) to a numpy array when doing evaluation w.r.t. a metric during the training?

body:
I am going through Andrew Ng’s tutorial from the CS230 Stanford course, and in every epoch of the training, evaluation is performed by calculating the metrics.
But before calculating the metrics, they are sending the batches to CPU and converting them to numpy arrays (code here).
# extract data from torch Variable, move to cpu, convert to numpy arrays
output_batch = output_batch.data.cpu().numpy()
labels_batch = labels_batch.data.cpu().numpy()

# compute all metrics on this batch
summary_batch = {metric: metrics[metric](output_batch, labels_batch) for metric in metrics}

My question is: why do they do that? Why don’t they just calculate the metrics (which is done here) on GPU using torch methods (e.g. torch.sum as opposed to np.sum)?
I would think that GPU to CPU transfers would slow things down, so there should be a very good reason for doing them?
I am new to PyTorch so I might be missing something.


title: GAN Model Summary Pytorch using TensorBoard?

body: Is there a way can I visualize the complete training loop for the GAN architecture in TensorBoard using Pytorch? I think it's possible using TF, but I am having a hard time to figure out one using Pytorch.


title: Train multiple models in pytorch simultaneously

body: I have k identical pytorch models, all of which can fit into a single GPU's memory.  I want to run a single training step on these models using k different input batches (one for each model).  I want to do it all a single call- ie I don't want to sequentially load them one at a time, but instead treat the k input batches as a single input to a single uber-model, generating a tensor of k losses, and a tensor of k gradients. Are there any abstractions available that would enable this?


title: looking for an equivalent of Tensorflow normalization layer in Pytorch

body:
I was using 'tf.keras.layers.experimental.preprocessing.Normalization'. This layer is cool since you can save weights in this layer to normalize any input data to this layer.
However, I couldn't find any normalization layer in Pytorch.
Is there a layer that functions the same role?


title: pytorch: Does variable.long() guarantee 64-bits?

body: In pytorch, I have a variable that might be IntTensor or cuda.IntTensor. It needs to be changed to 64-bits retaining cpu/gpu. Does variable.long() guarantee 64-bits on all implementations of pytorch? If not, how can variable be converted to 64-bits retaining cpu/gpu for all implementations?


title: RuntimeError when changing the values of specific parts of a `torch.Tensor`

body:
Say I have a 3 dimentional tensor x initialized with zeros:

x = torch.zeros((2, 2, 2))

and an other 3 dimentional tensor y

y = torch.ones((2, 1, 2))

I am trying to change the values of the first line of x[0] and x[1] like this

x[:, 0, :] = y

but I get this error:

RuntimeError: expand(torch.FloatTensor{[2, 1, 2]}, size=[2, 2]): the number of sizes provided (2) must be greater or equal to the number of dimensions in the tensor (3)

It is as if the tensor y was getting squeezed somehow.  Is there a way around this?


title: Is .data still useful in pytorch 1.3 stable and what is the meaning of it?

body:
Is .data still be used in the pytorch 1.3 stable, if so, could you please share the reference to me? Thx.

t = torch.randperm(8)
t.data


title: Neural network versus random forest performance discrepancy

body:
I want to run some experiments with neural networks using PyTorch, so I tried a simple one as a warm-up exercise, and I cannot quite make sense of the results.

The exercise attempts to predict the rating of 1000 TPTP problems from various statistics about the problems such as number of variables, maximum clause length etc. Data file https://github.com/russellw/ml/blob/master/test.csv is quite straightforward, 1000 rows, the final column is the rating, started off with some tens of input columns, with all the numbers scaled to the range 0-1, I progressively deleted features to see if the result still held, and it does, all the way down to one input column; the others are in previous versions in Git history.

I started off using separate training and test sets, but have set aside the test set for the moment, because the question about whether training performance generalizes to testing, doesn't arise until training performance has been obtained in the first place.

Simple linear regression on this data set has a mean squared error of about 0.14.

I implemented a simple feedforward neural network, code in https://github.com/russellw/ml/blob/master/test_nn.py and copied below, that after a couple hundred training epochs, also has an mean squared error of 0.14.

So I tried changing the number of hidden layers from 1 to 2 to 3, using a few different optimizers, tweaking the learning rate, switching the activation functions from relu to tanh to a mixture of both, increasing the number of epochs to 5000, increasing the number of hidden units to 1000. At this point, it should easily have had the ability to just memorize the entire data set. (At this point I'm not concerned about overfitting. I'm just trying to get the mean squared error on training data to be something other than 0.14.) Nothing made any difference. Still 0.14. I would say it must be stuck in a local optimum, but that's not supposed to happen when you've got a couple million weights; it's supposed to be practically impossible to be in a local optimum for all parameters simultaneously. And I do get slightly different sequences of numbers on each run. But it always converges to 0.14.

Now the obvious conclusion would be that 0.14 is as good as it gets for this problem, except that it stays the same even when the network has enough memory to just memorize all the data. But the clincher is that I also tried a random forest, https://github.com/russellw/ml/blob/master/test_rf.py

... and the random forest has a mean squared error of 0.01 on the original data set, degrading gracefully as features are deleted, still 0.05 on the data with just one feature.

Nowhere in the lore of machine learning is it said 'random forests vastly outperform neural nets', so I'm presumably doing something wrong, but I can't see what it is. Maybe it's something as simple as just missing a flag or something you need to set in PyTorch. I would appreciate it if someone could take a look.

import numpy as np
import pandas as pd
import torch
import torch.nn as nn

# data
df = pd.read_csv("test.csv")
print(df)
print()

# separate the output column
y_name = df.columns[-1]
y_df = df[y_name]
X_df = df.drop(y_name, axis=1)

# numpy arrays
X_ar = np.array(X_df, dtype=np.float32)
y_ar = np.array(y_df, dtype=np.float32)

# torch tensors
X_tensor = torch.from_numpy(X_ar)
y_tensor = torch.from_numpy(y_ar)

# hyperparameters
in_features = X_ar.shape[1]
hidden_size = 100
out_features = 1
epochs = 500

# model
class Net(nn.Module):
    def __init__(self, hidden_size):
        super(Net, self).__init__()
        self.L0 = nn.Linear(in_features, hidden_size)
        self.N0 = nn.ReLU()
        self.L1 = nn.Linear(hidden_size, hidden_size)
        self.N1 = nn.Tanh()
        self.L2 = nn.Linear(hidden_size, hidden_size)
        self.N2 = nn.ReLU()
        self.L3 = nn.Linear(hidden_size, 1)

    def forward(self, x):
        x = self.L0(x)
        x = self.N0(x)
        x = self.L1(x)
        x = self.N1(x)
        x = self.L2(x)
        x = self.N2(x)
        x = self.L3(x)
        return x

model = Net(hidden_size)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

# train
print("training")
for epoch in range(1, epochs + 1):
    # forward
    output = model(X_tensor)
    cost = criterion(output, y_tensor)

    # backward
    optimizer.zero_grad()
    cost.backward()
    optimizer.step()

    # print progress
    if epoch % (epochs // 10) == 0:
        print(f"{epoch:6d} {cost.item():10f}")
print()

output = model(X_tensor)
cost = criterion(output, y_tensor)
print("mean squared error:", cost.item())


title: How to use trained deep learning model to predict in excel?

body:
After we trained my neural network and saved my model with tensorflow, we can load the model and predict result like following:

from keras.models import load_model
model = load_model('my_model.h5')
result = model.predict(test_input)

Is there any way we can use the trained model in excel to do the similar job? I assume we can build a excel user defined function to take n parameter and give a number. Can we do in VBA? How to load the model file and run the predict?


title: Understanding batch_size in CNNs

body:
Say that I have a CNN model in Pytorch and 2 inputs of the following sizes:

input_1: [2, 1, 28, 28]
input_2: [10, 1, 28, 28]

Notes:

To reiterate, input_1 is batch_size == 2 and input_2 is batch_size == 10.
Input_2 is a superset of input_1. That is, input_2 contains the 2 images in input_1 in the same position.

My question is: how does the CNN process the images in both inputs? I.e. does the CNN process every image in the batch sequentially? Or does it concatenate all of the images in the batch size and then perform convolutions per usual?

The reason I ask is because:

The output of CNN(input_1) != CNN(input_2)[:2]

That is, the difference in batch_size results in slightly different CNN outputs for both inputs for the same positions.


title: My PyTorch Conv1d with an identity kernel does not produce the same output as the input

body:
I have:

        print('\ninp', inp.min(), inp.mean(), inp.max())
        print(inp)
        out = self.conv1(inp)
        print('\nout1', out.min(), out.mean(), out.max())
        print(out)
        quit()

My min, mean and max for my inp is: inp tensor(9.0060e-05) tensor(0.1357) tensor(2.4454)

For my output, I have: out1 tensor(4.8751, grad_fn=<MinBackward1>) tensor(21.8416, grad_fn=<MeanBackward0>) tensor(54.9332, grad_fn=<MaxBackward1>)

My self.conv1 is:

        self.conv1 = torch.nn.Conv1d(
            in_channels=161,
            out_channels=161,
            kernel_size=11,
            stride=1,
            padding=5)
        self.conv1.weight.data = torch.zeros(self.conv1.weight.data.size())
        self.conv1.weight.data[:, :, 5] = 1.0
        self.conv1.bias.data = torch.zeros(self.conv1.bias.data.size())

So my weights look like: tensor([0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.])

So if I understand how convolution works, this should produce the same output. But it doesn't.

What am I doing wrong?


title: Keras vs Pytorch NN code small differences, need clarification

body:
I have the Keras and Pytorch code for the same neural network. Some of the lines are switched around between the two.
I am wondering why for the Pytorch version max pooling comes before batch normalization and reel activation. In Keras it comes after those two lines. And for flattening, I'm also confused on how Pytorch used 64 * 7 * 7 (where do the 7s come from?).

Here's the Keras version of the Shallow net Alex net:

def shallownet(nb_classes):
    global img_size
    model = Sequential()
    model.add(Conv2D(64, (5, 5), input_shape=img_size, data_format='channels_first'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same', data_format='channels_first'))

    model.add(Conv2D(64, (5, 5), padding='same', data_format='channels_first'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same', data_format='channels_first'))

    model.add(Flatten())
    model.add(Dense(384))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(192))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(nb_classes, activation='softmax'))
    return model

and the Pytorch version:

class AlexNet(nn.Module):

    def __init__(self, num_classes=10):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=5, padding=2,
                      bias=False),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Conv2d(64, 64, kernel_size=5, padding=2, bias=False),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
        )
        self.classifier = nn.Sequential(
            nn.Linear(64 * 7 * 7, 384, bias=False),
            nn.BatchNorm1d(384),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(384, 192, bias=False),
            nn.BatchNorm1d(192),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(192, num_classes)
        )
        self.regime = {
            0: {'optimizer': 'SGD', 'lr': 1e-3,
                'weight_decay': 5e-4, 'momentum': 0.9},
            60: {'lr': 1e-2},
            120: {'lr': 1e-3},
            180: {'lr': 1e-4}
        }

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, 64 * 7 * 7)
        x = self.classifier(x)
        return F.log_softmax(x)

def cifar10_shallow(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 10)
    return AlexNet(num_classes)

def cifar100_shallow(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 100)
    return AlexNet(num_classes)


title: Pytorch GPU utlilization

body:
So I'm currently training my DQN on Atari Breakout (from OpenAI gym). The problem I'm having is that it is very slow, but my GPU does not seem to be fully utilized. I'm  monitoring it with a tool called HWMonitor and also with the nvidia-smi command. Any ideas why?
Here are some specs that might be relevant:
Python version: 3.7.7
torch version: 1.4.0
gym version: 0.17.2
CUDA version: 11.0
GPU: GTX 980 founders edition
CPU: Inte Core i7 4790K

And here is some insight into my DQN:
device:cuda
DQN(
  (conv1): Conv2d(1, 16, kernel_size=(8, 8), stride=(4, 4))
  (conv2): Conv2d(16, 32, kernel_size=(4, 4), stride=(2, 2))
  (mp1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (mp2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (lin1): Linear(in_features=128, out_features=256, bias=True)
  (lin2): Linear(in_features=256, out_features=4, bias=True)
)
optimizer: Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    lr: 0.001
    weight_decay: 0
)
gamma: 0.95
epsilon_decay: 0.99
epsilon_min: 0.01
obs_space: [110, 84]
trainig_batchsize: 32

Here you can see how my monitoring looks while training with the GPU:

One more thing that might be important. For some reason training with the CPU is faster then with the GPU. These results are an average over 30 minutes of training. I can't explain why my CPU is faster so if anybody has a clue, I would definitely like to hear it.
GPU took 353 seconds for 1000 minibatches
CPU took 148 seconds for 1000 minibatches

If any of my code is needed to further inspect my problem please let me know.
EDIT:
I'm now thinking of using cloud GPU services, more specific Linode. The only thing that worries me is that the cause of my problem lies in my code. Thus using a cloud GPU service would not be very smart.


title: How to avoid overfitting in deep learning when features are binary in nature

body:
I am constructing a deep learning model using 2048 bits of binary fingerprints (0 and 1's) for some 2000 samples to predict their outputs (positive (1) OR negative(0)). The feature data is quite sparse i.e. lots of zeros and rare 1's.
I have used 'binary cross entropy' but my validation accuracy doesn't increase more than 70%. I have balanced data. The model seems to be overfitting. I can't normalize my data since fetures are binary. How can I avoid overfitting?
    earlystop = EarlyStopping(monitor='val_acc', patience=20, mode='max')
    final_model = Sequential()
    final_model.add(Dense(1012, input_dim=2048, activation = 'relu'))
    final_model.add(BatchNormalization())
    final_model.add(Dropout(0.9))
    final_model.add(Dense(512, activation='relu'))
    final_model.add(BatchNormalization())
    final_model.add(Dropout(0.9))
    final_model.add(Dense(128, activation='relu'))
    final_model.add(BatchNormalization())
    final_model.add(Dropout(0.5))
    final_model.add(Dense(32, activation='relu'))
    final_model.add(Dense(1, activation='sigmoid'))
    adam_opt = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
    final_model.compile(loss='binary_crossentropy', optimizer=adam_opt, metrics=['accuracy'])


title: Loading images in PyTorch

body:
I am new to PyTorch and working on a GAN model. I want to load my image dataset. The way its done using Keras is:
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import load_img

def load_images(path, size=(128,128)):
data_list = list()
# enumerate filenames in directory, assume all are images
for filename in listdir(path):
    # load and resize the image
    pixels = load_img(path + filename, target_size=size)
    # convert to numpy array
    pixels = img_to_array(pixels)
    # store.
    data_list.append(pixels)
return asarray(data_list)
# dataset path
path = 'mypath/'
# load dataset A
dataA = load_images(path + 'A/')
dataAB = load_images(path + 'B/')

I want to know how to do the same in PyTorch.
Any help is appreciated. Thanks


title: PyTorch autograd: dimensionality of custom function gradients?

body:
Question summary: How is the dimensionality of inputs and outputs handled in the backward pass of custom functions?
According to the manual, the basic structure of custom functions is the following:
class MyFunc(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input): # f(x) = e^x
        result = input.exp()
        ctx.save_for_backward(result)
        return result

    @staticmethod
    def backward(ctx, grad_output): # df(x) = e^x
        result, = ctx.saved_tensors
        return grad_output * result

For a single input and output dimension, this is perfectly fine and works like a charm. But for higher dimensions the backward pass becomes confusing. Apparently, PyTorch only accepts a result of backward that has the same dimensionality as the result of forward (for the same input). Returning a wrong shape yields a RuntimeError: Function MyFunc returned an invalid gradient at index 0 - got [*] but expected shape compatible with [*]. So I am wondering: What does backward actually compute?
Its not a Jacobian? For example, when I have a function f(x) = ( f_1(x_1, ... , x_n), ... , f_k(x_1, ... , x_n) ) with n inputs and k outputs, I would expect that a gradient calculation would yield a Jacobian matrix of dimension k*n. However, the PyTorch implementation expects just a vector of dimension n. So what does the backward result actually mean, it can't be the Jacobian?
And it does not handle batches? Moreover, what if I would like to push a batch of input vectors through this function, e.g. an input of dimension b*n with batch size b. Then, instead of something like b*k*n the gradient is expected to also have the shape b*n. Is it even intended to consider the processing of batches with custom functions?
None of these questions seems to be addressed in the manual and the provided examples are very simple, which does not help at all. Maybe there are formulas hidden somewhere that explain the background of the provided Function interface in more detail, but I haven't found them yet.


title: Transformer for time series forecasting

body:
I have discrete daily features and a target time series and I'm trying to implement a basic Transformer for seq2seq modeling. I construct my supervised data as follows:
 . . .\. . .\. . . .
       |
\0 0 0 0\0 0 0 0 0 0

The next sequence is shifted by one position ahead. Overall I have the input data shape [batch_size, in_sequence_len, num_features] and the target is [batch_size, out_sequence_len, 1]. I understand that encoder input should be of shape [batch_size, in_seq_len, num_features] and decoder takes [batch_size, out_seq_len, num_features]. But how should I transform my batch to be suitable for the transformer input?


title: torch.addmm received an invalid combination of arguments

body:
In the official webpage of pytorch I saw the following code with answers:

>> a = torch.randn(4, 4)
>> a

0.0692  0.3142  1.2513 -0.5428
0.9288  0.8552 -0.2073  0.6409
1.0695 -0.0101 -2.4507 -1.2230
0.7426 -0.7666  0.4862 -0.6628
torch.FloatTensor of size 4x4]

>>> torch.max(a, 1)
(
 1.2513
 0.9288
 1.0695
 0.7426
[torch.FloatTensor of size 4]
,
 2
 0
 0
 0
[torch.LongTensor of size 4]
)

I know that the first result corresponds to the maximum number per row, however I do not get the second tensor (LongTensor)

I tried other random example and after a pytorch.max, I came to find these results

0.9477  1.0090  0.8348 -1.3513
-0.4861  1.2581  0.3972  1.5751
-1.2277 -0.6201 -1.0553  0.6069
 0.1688  0.1373  0.6544 -0.7784
[torch.FloatTensor of size 4x4]

(
 1.0090
 1.5751
 0.6069
 0.6544
[torch.FloatTensor of size 4]
,
 1
 3
 3
 2
[torch.LongTensor of size 4]
)

Can anyone tell me what does it really mean these LongTensor data? I thought it was a strange casting between tensors, however after a simple casting of a float tensor, I see that it just cuts decimals

Thanks


title: Getting multiprocessing lock error when running vizdoom and pytorch program on Windows Subsystem for Linux

body:
whenever I try to run my  program on WSL, I get the following error. I'm pretty new to pytorch and vizdoom, so I don't know how to solve this problem.

Setup
- Windows 10 x64
- Ubuntu 14 (on WSL)
- Python 2.7.14  (Anaconda 2)
- OpenAI Gym 0.9.5
- Vizdoom 1.1.4
- doom-py 0.0.14
- ppaquette/gym-doom
- pytorch 0.0.12

(doomenv) hybridsyntax@Blacklynx:/mnt/f/_TUTORIALS/ai/doom/code$ python ai.py  > wsl.log
[2018-05-25 18:21:44,354] Making new env: ppaquette/DoomCorridor-v0
[2018-05-25 18:21:44,365] Clearing 2 monitor files from previous run (because force=True was provided)
Assertion 'pthread_mutex_unlock(&m->mutex) == 0' failed at pulsecore/mutex-posix.c:108, function pa_mutex_unlock(). Aborting.
Traceback (most recent call last):
  File "ai.py", line 155, in <module>
    memory.run_steps(200)
  File "/mnt/f/_TUTORIALS/ai/doom/code/experience_replay.py", line 70, in run_steps
    entry = next(self.n_steps_iter) # 10 consecutive steps
  File "/mnt/f/_TUTORIALS/ai/doom/code/experience_replay.py", line 21, in __iter__
    state = self.env.reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/wrappers/monitoring.py", line 39, in _reset
    observation = self.env.reset(**kwargs)
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 311, in _reset
    observation = self.env.reset(**kwargs)
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/wrappers/frame_skipping.py", line 33, in _reset
    return self.env.reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 283, in _reset
    return self.env.reset(**kwargs)
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/wrappers/time_limit.py", line 49, in _reset
    return self.env.reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym-doom/ppaquette_gym_doom/doom_env.py", line 244, in _reset
    return self._load_level()
  File "/mnt/f/_TUTORIALS/ai/doom/gym-doom/ppaquette_gym_doom/doom_env.py", line 157, in _load_level
    'singleton lock in memory.')
gym.error.Error: [ViZDoomUnexpectedExitException, ViZDoomErrorException] VizDoom exited unexpectedly. This is likely caused by a missing multiprocessing lock. To run VizDoom across multiple processes, you need to pass a lock when you configure the env [e.g. env.configure(lock=my_multiprocessing_lock)], or create and close an env before starting your processes [e.g. env = gym.make("DoomBasic-v0"); env.close()] to cache a singleton lock in memory.
[2018-05-25 18:21:44,696] Finished writing results. You can upload them to the scoreboard via gym.upload('/mnt/f/_TUTORIALS/ai/doom/code/videos')
(doomenv) hybridsyntax@Blacklynx:/mnt/f/_TUTORIALS/ai/doom/code$

Thanks in advance


title: Can I sum the gradients in each training iteration?

body:
I need to sum the gradients in each iteration and then transfer these gradients to another process to reproduce the learned network.

The key code is like this below. Method 1:

class Net(nn.Module):
    def __init__(self, state_dim, action_dim, max_action):
        super(Actor, self).__init__()

        self.l1 = nn.Linear(state_dim, 40)
        self.l2 = nn.Linear(40, 30)
        self.l3 = nn.Linear(30, action_dim)

        self.max_action = max_action

    def forward(self, x):
        x = F.relu(self.l1(x))
        x = F.relu(self.l2(x))
        x = self.max_action * torch.tanh(self.l3(x))
        return x

def train(batches,state_dim, action_dim, max_action):
    actor = Net(state_dim, action_dim, max_action)
    critic = Net(state_dim, action_dim, max_action)

    for i in range(1000):
        ...
        #Compute critic loss
        critic_loss = F.mse_loss(current_Q, target_Q)

        # Optimize the critic
        critic_optimizer.zero_grad()
        critic_loss.backward()
        critic_optimizer.step()

        # Compute actor loss
        actor_loss = -critic(state,actor(state)).mean()

        # Optimize the actor
        actor_optimizer.zero_grad()
        actor_loss.backward()
        actor_optimizer.step()

    return net

...
net = train(batches,state_dim, action_dim, max_action)

Method 2:

...

def train(batches,state_dim, action_dim, max_action):
    net = Net(state_dim, action_dim, max_action)
    for i in range(1000):
        ...

        # Optimize the critic
        critic_optimizer.zero_grad()
        critic_loss.backward()
        sum_grads(critic) # sum the gradient in critic

    for g,p in zip(sum_grads,net.parameters()):
        p.grad = torch.from_numpy(g)
    net_optimizer.step()

    return net

...
net = train(batches,state_dim, action_dim, max_action)

I hope method 1 and method two can learn the same net parameters, but it didn't. So my question is why? And how to make it work? Thank you in advance.


title: Can't load weights-yolo weight

body:
I want to load pretrained yolo weights in a .weights file with pytorch . I entire the whole path way to .weight file but across an error.

model = load_model(weights='C:/Users/.../yolov2-tiny-voc_2.weights')
print(model)

UnpicklingError                           Traceback (most recent call last)

UnpicklingError: invalid load key, '\x00'.


title: mean[:, None, None] crashes in Pytorch / jupyter

body:
I previously had this "data_transforms" that I was using to construct my dataloader :

data_transforms_train = transforms.Compose([
    transforms.Resize((512, 512)),
    transforms.RandomVerticalFlip(),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor()
])

Everything was working fine and I could train my model without any problem.
I decided to include a normalization step at the end of the transformations:

data_transforms_train = transforms.Compose([
    transforms.Resize((512, 512)),
    transforms.RandomVerticalFlip(),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.8979], std=[0.3025])
])

Adding this started caused my notebook to crash. In other words, I didn't get any python message error but the notebook kernel kernel kept crashing and restarting.
At first I thought it was a memory error so I changed my batch_size to 1 to verify but it wasn't the problem.

I tried to inspect so I applied the transformations one by one and copied the "to_normalize" source code from https://pytorch.org/docs/stable/_modules/torchvision/transforms/functional.html

training_set = Dataset(partition['train'], labels, transform=None)
d = transforms.functional.resize(training_set.__getitem__(3)[0], (512,512))
x = transforms.functional.to_tensor(d)
# Normalization code
mean, std = [0.7], [0.3025]
mean, std = torch.as_tensor(mean, dtype=x.dtype, device=x.device), torch.as_tensor(std, dtype=x.dtype, device=x.device)

So far so good. When I add this :

tensor.sub_(mean[:, None, None]).div_(std[:, None, None])

The notebook crashes again (no python error, just the kernel crashing and restarting). When I isolated the parts, it turned out that mean[:, None, None] (or/also std[:, None, None]) causes the notebook to crash.

Does anyone have any idea why is that happening ?


title: convLSTM size mismatch

body:
I'm trying to implement just one cell of Convolutional LSTM and passing a tensor (1,3,128,128) in it. I get a size mismatch error.

    class ConvLSTMCell(nn.Module):
        def __init__(self, input_size, input_dim, hidden_dim, kernel_size, bias):
            """
            Parameters
            ----------
            input_size: (int, int)
                Height and width of input tensor as (height, width).
            input_dim: int
                Number of channels of input tensor.
            hidden_dim: int
                Number of channels of hidden state.
            kernel_size: (int, int)
                Size of the convolutional kernel.
            bias: bool
                Whether or not to add the bias.
            """
            super(ConvLSTMCell, self).__init__()

            self.height, self.width = input_size
            self.input_dim  = input_dim
            self.hidden_dim = hidden_dim

            self.kernel_size = kernel_size
            # self.padding     = kernel_size[0] // 2, kernel_size[1] // 2
            self.bias        = bias

            self.conv = nn.Conv2d(in_channels=self.input_dim + self.hidden_dim,
                              out_channels=4 * self.hidden_dim,
                              kernel_size=self.kernel_size,
                              #padding=self.padding,
                              bias=self.bias)

        def forward(self, input, prev_state):
            h_prev, c_prev = prev_state
            print('x: {}\nh_prev: {}\nc_prev: {}'.format(x.size(), h_prev.size(), c_prev.size()))
            combined = torch.cat((input, h_prev), dim=1) # concatenate along channel axis
            print('combined: {}'.format(combined.size()))

            combined_conv = self.conv(combined)
            print('combined_conv: {}'.format(combined_conv.size()))
            cc_i, cc_f, cc_o, cc_g = torch.split(combined_conv, self.hidden_dim, dim=1)
            print('cc_i: {}\ncc_f: {}\ncc_o: {}\ncc_g: {}'.format(cc_i.size(), cc_f.size(),   cc_o.size(), cc_g.size()))

            i = torch.sigmoid(cc_i)
            f = torch.sigmoid(cc_f)
            o = torch.sigmoid(cc_o)
            g = torch.tanh(cc_g)
            print('i: {}\nf: {}\no: {}\ng: {}'.format(i.size(), f.size(), o.size(), g.size()))

            c_cur = f * c_prev + i * g
            h_cur = o * F.tanh(c_cur)
            print('c_cur: {}\nh_cur: {}'.format(c_cur.size(), h_cur.size()))

            return h_cur, c_cur

        def init_hidden(self, batch_size):
            return (Variable(torch.zeros(batch_size, self.hidden_dim, self.height, self.width)),
                    Variable(torch.zeros(batch_size, self.hidden_dim, self.height, self.width)))

    x = torch.randn(1,3,128,128)
    model = ConvLSTMCell(input_size=(128,128), input_dim=3, hidden_dim=3, kernel_size=(5,5),
                         bias=True)
    hc = model.init_hidden(batch_size=1)
    if gpu:
        x.cuda()
        model.cuda()
        hc.cuda()

    out = model(x, hc)
    print(out.size())

I get the following error:

x: torch.Size([1, 3, 128, 128])
h_prev: torch.Size([1, 3, 128, 128])
c_prev: torch.Size([1, 3, 128, 128])
combined: torch.Size([1, 6, 128, 128])
combined_conv: torch.Size([1, 12, 124, 124])
cc_i: torch.Size([1, 3, 124, 124])
cc_f: torch.Size([1, 3, 124, 124])
cc_o: torch.Size([1, 3, 124, 124])
cc_g: torch.Size([1, 3, 124, 124])
i: torch.Size([1, 3, 124, 124])
f: torch.Size([1, 3, 124, 124])
o: torch.Size([1, 3, 124, 124])
g: torch.Size([1, 3, 124, 124])
Traceback (most recent call last):
File "trial.py", line 87, in
out = model(x, hc)
File "/Users/abcde/opt/anaconda3/envs/matrix/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "trial.py", line 66, in forward
c_cur = f * c_prev + i * g
RuntimeError: The size of tensor a (124) must match the size of tensor b (128) at non-singleton >dimension 3

I'm wish to build a network of 17 cells using this and I want to use the output of every cell to calculate loss against the ground truth. Ground truth are 18 (3,128,128) images.
How do I get my network to output hidden states of the same size?


title: Repeat specific columns of a tensor in Pytorch

body:
I have a pytorch tensor X of size m x n  and a list of nonnegative integers num_repeats of length n (assume sum(num_repeats)>0). Inside a forward() method, I want to create a tensor X_dup of size m x sum(num_repeats) where column i of X is repeated  num_repeats[i] times. The tensor X_dup is to be used downstream in the forward() method so the gradient needs to be backpropogated correctly.
All solutions I could come up with require inplace operations (creating a new tensor and populating it by iterating over  num_repeats), but if I understand correctly this won't preserve the gradient (please correct me if I'm wrong, I'm new to the whole Pytorch thing).


title: Using Torch for numerical optimization, getting optimizer got an empty parameter list,

body:
I am doing some computations and would like to optimise the parameters of this by using Pytorch. I am NOT defining a neural network, so no layers and stuff like that. Just a simple sequence of computations. I use the torch.nn.Module to be able to use Pytorch's optimisers.

My class looks something like this:

class XTransformer(torch.nn.Module):

    def __init__(self, x):
        super(ReLUTransformer, self).__init__()
        self.x = x

    def funky_function(self, m, c):

        # do some computations
        m = self.x * 2 - m + c

        return m, c

    def forward(self, m, c):
        m, c = self.funky_function(m, c)
        return m, c

Later on I define and try to optimise this parameter x like so:

x = torch.autograd.Variable(x, requires_grad=True)
model = XTransformer(x)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
m, c = smt
loss = loss_func()

for t in range(100):
    m , c = model(m, c)
    l = loss(m, true)
    optimizer.zero_grad()
    l.backward()
    optimizer.step()

I don't know what to do. I get the "ValueError: optimizer got an empty parameter list" error. when I just give [x] as an argument to the optimizer, It doesn't update and change x for me. What should I do?


title: Plot several graphs in Pytorch tensorboard

body:
I am training a dynamic neural network, meaning that each epoch I tweak the architecture and get a different computational graph.
I want to plot the graph for each epoch using tensorboard, but when I use SummaryWriter.add_graph() at the end of each epoch it simply overwrites the previous one.

Any ideas how to plot several graphs using pytorch + tensorboard? It seems achievable as each graph has a “tag” but I found no option to change this tag to plot several of them.

Thanks,
Elad


title: How to convert PyTorch Tensor to Numpy ndarray

body:
I have this function that rotates the MNIST images. The function returns a pytorch Tensor. I am more familiar with Tensorflow and I want to convert the pytorch tensor to a numpy ndarray that I can use. Is there a function that will allow me to do that? I tried to modify the function a little bit by adding .numpy() after tensor(img.rotate(rotation)).view(784) and save it in an empty ndarray, but that didn't work. Parameter d is MNIST data saved in .pt (pytensor, I think). Thanks! (Would love to know if there is a tensorflow function that can rotate the data.)

t = 1
min_rot = 1.0 * t / 20 * (180 - 0) + \
        0
max_rot = 1.0 * (t + 1) / 20 * \
    (180 - 0) + 0
rot = random.random() * (max_rot - min_rot) + min_rot
rotate_dataset(x_tr, rot)

def rotate_dataset(d, rotation):
    result = torch.FloatTensor(d.size(0), 784)
    tensor = transforms.ToTensor()

    for i in range(d.size(0)):
        img = Image.fromarray(d[i].numpy(), mode='L')
        result[i] = tensor(img.rotate(rotation)).view(784)
    return result


title: Minimal (light version) PyTorch and Numpy packages in production

body:
I am putting a model into production and I am required to scan all dependencies (Pytorch and Numpy) beforehand via VeraCode Scan.

I noticed that the majority of the flaws are coming from test scripts and caffe2 modules in Pytorch and numpy.

Is there any way to build/install only part of these packages that I use in my application? (e.g. I won't use testing and caffe2 in the application so there's no need to have them in my PyTorch / Numpy source code)


title: calculating accuracy for Monte carlo Dropout on pytorch

body:
I have found an implementation of the Monte carlo Dropout on pytorch the main idea of implementing this method is to set the dropout layers of the model to train mode. This allows for different dropout masks to be used during the different various forward passes.
The implementation illustrate how multiple predictions from the various forward passes are stacked together and used for computing different uncertainty metrics.
import sys

import numpy as np

import torch
import torch.nn as nn

def enable_dropout(model):
    """ Function to enable the dropout layers during test-time """
    for m in model.modules():
        if m.__class__.__name__.startswith('Dropout'):
            m.train()

def get_monte_carlo_predictions(data_loader,
                                forward_passes,
                                model,
                                n_classes,
                                n_samples):
    """ Function to get the monte-carlo samples and uncertainty estimates
    through multiple forward passes

    Parameters
    ----------
    data_loader : object
        data loader object from the data loader module
    forward_passes : int
        number of monte-carlo samples/forward passes
    model : object
        keras model
    n_classes : int
        number of classes in the dataset
    n_samples : int
        number of samples in the test set
    """

    dropout_predictions = np.empty((0, n_samples, n_classes))
    softmax = nn.Softmax(dim=1)
    for i in range(forward_passes):
        predictions = np.empty((0, n_classes))
        model.eval()
        enable_dropout(model)
        for i, (image, label) in enumerate(data_loader):

            image = image.to(torch.device('cuda'))
            with torch.no_grad():
                output = model(image)
                output = softmax(output) # shape (n_samples, n_classes)
            predictions = np.vstack((predictions, output.cpu().numpy()))

        dropout_predictions = np.vstack((dropout_predictions,
                                         predictions[np.newaxis, :, :]))
        # dropout predictions - shape (forward_passes, n_samples, n_classes)

    # Calculating mean across multiple MCD forward passes
    mean = np.mean(dropout_predictions, axis=0) # shape (n_samples, n_classes)

    # Calculating variance across multiple MCD forward passes
    variance = np.var(dropout_predictions, axis=0) # shape (n_samples, n_classes)

    epsilon = sys.float_info.min
    # Calculating entropy across multiple MCD forward passes
    entropy = -np.sum(mean*np.log(mean + epsilon), axis=-1) # shape (n_samples,)

    # Calculating mutual information across multiple MCD forward passes
    mutual_info = entropy - np.mean(np.sum(-dropout_predictions*np.log(dropout_predictions + epsilon),
                                            axis=-1), axis=0) # shape (n_samples,)

What I'm trying to do is to calculate accuracy across different forward passes, can anyone please help me on how  to get the accuracy and make any changes on the dimensions used in this implementation
I am using the CIFAR10 dataset and would like to use the dropout on test time The code for the data_loader
 testset = torchvision.datasets.CIFAR10(root='./data', train=False,download=True, transform=test_transform)

 #loading the test set
data_loader = torch.utils.data.DataLoader(testset, batch_size=n_samples, shuffle=False, num_workers=4) ```


title: How to use a numpy-based external library function within a Tensorflow 2/keras custom layer?

body:
I would be very grateful if someone would help me understanding the following problem.
I am trying to implement a custom layer in tensorflow 2 by using keras (it's a layer derived from the class Layer).
I have overriden the build and the call functions.
When I write my call function I need to invoke a method from an external library which only accepts numpy array. It's a pretty complex function and of course it does not use tensorflow functions.
My call function takes in input a tensor, convert it to numpy (by .numpy() function), invoke the external method from another library, then converts the numpy array back to tensor.
Here I have some issues and tried different solutions.

If I run the code it tells me that ...ops.Tensor has no method .numpy(). If I correctly understood, this is due to the type of tensor I receive in input to the call function and I need to use the eager mode.
If I compile by setting the run_eagerly flag to true, during the fit, it tells me that the gradient is missing. I was hoping that some form of auto-grad computation was implemented but maybe this is not what it is currently happening?.

Source code:
class SimpleLayer(tf.keras.layers.Layer):
    def __init__(self):
        super(SimpleLayer, self).__init__()

    def build(self, input_shape):
        shape_a = input_shape[0]                #batch_size
        shape_b = int(input_shape[1])           #height
        shape_c = int(input_shape[2])           #width
        shape_d = input_shape[3]                #channel_number
        self.w = self.add_weight(shape=(shape_d*shape_a, 2, shape_b, shape_c),
                                 initializer='random_uniform', dtype='float64',
                                 trainable=True)

    def call(self, inputs):
        tf_inputs = inputs.numpy()
        sft = tr.transform(tf_inputs) # call to the external library
        transformed = tf.math.multiply(sft, self.w)
        conv = tf.zeros(np.shape(inputs))
        conv = tr.inv_transform(inputs, transformed) # another call to a function that also converts to tensor
        return conv. # this is a tensor!

.....

model = tf.keras.Sequential([
    layers.experimental.preprocessing.Rescaling(1./255),
    pl.SimpleLayer(),
    layers.Activation('relu'),
    layers.MaxPooling2D(),
    pl.SimpleLayer(),
    layers.Activation('relu'),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dense(num_classes)
])

model.compile(
    optimizer='adam',
    loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
    run_eagerly=True,  # adding this to avoid graph building
    metrics=['accuracy'])

history = model.fit(train_images, train_labels, epochs=3, batch_size=20,
                    validation_data=(test_images, test_labels))

Is there any workaround for that? Can I only use tensorflow 2 functions to process the tensor or maybe I am missing something?
I am also considering to switch to pyTorch (which I do not know yet) but I fear to have the same issue. What would you suggest?


title: How to get class and bounding box coordinates from YOLOv5 predictions?

body:
I am trying to perform inference on my custom YOLOv5 model. The official documentation uses the default detect.py script for inference. I have written my own python script but I cannot access the predicted class and the bounding box coordinates from the output of the model. Here is my code:
import torch

model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='best.pt')
predictions = model("my_image.png")

print(predictions)


title: Pytorch LongTensor of large integer gives float float value

body:
x = torch.LongTensor([108500])
print (x)

gives

1.0850e+05
[torch.LongTensor of size 1]

Why is it so? Can't we have LongTensor of large integer?
How can I convert the value of x to be integer so that I can get the embedding of x.


title: Pytorch resumes training after every training session

body:
I have a dataset which is partitioned into smaller datasets.

I want to train 3 models for each partition of the dataset, but I need all training sessions to start from the same initialised network parameters.

so it looks like this:

modelList = []
thisCNN = NNet()

for x in range(3):
    train = torch.utils.data.DataLoader(Subset(train_set, indexes[x]), batch_size=32)
    bb = trainMyNet(thisCNN, train, test)
    modelList.append(list(bb.parameters()))

print modelList[0][1]
print modelList[1][1]
print modelList[2][1]

In the printing, I am getting the same exact parameters for every saved model which is strange, and also I noticed that after every iteration the model in fact resumes the training from the previous iteration as it results in lower loss per iteration.

What I am trying to achieve, is, per iteration to get a new model on the current subset x but the training should start with the same initial thisCNN = NNet() weights.


title: tensorflow equivalent of pytorch ReplicationPad2d

body:
I'm trying to figure out how to do the tensorflow equivalent of the following padding in pytorch:

nn.ReplicationPad2d((1, 0, 1, 0))

I've tried the following, but this only seems to work if the input tensor is actually 2x2:

tf.pad(my_tensor, [[1, 0], [1, 0]], "SYMMETRIC")


title: Unable to use backward hooks in a loop

body:
I am trying to implement The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks in pytorch using backward hooks to stop incoming gradient for freezing some weights. Everything seems to work fine when adding the hooks are added manually but it shows an error when I try to do the same in a loop. The mask_dict is a dictionary with layer name as keys and boolean mask as values. The mask is created by setting the lowest pr_value% l1-norm of weights to be zero. Below is the code

def l1norm_mask(model, pr_value, prune_large=False):
    mask_dict={}
    for name, module in model.named_modules():
        if 'linear' in name:
            t = module.weight
            bottomk = torch.topk(torch.abs(t).view(-1), k=round(pr_value*t.nelement()), largest=prune_large)
            clone = t.clone().detach()
            clone.view(-1)[bottomk.indices] = 0
            mask_dict[name] = clone.bool()
    return mask_dict

def apply_mask(model, mask_dict):
    for name, module in model.named_modules():
        if 'linear' in name:
            print('')
            module.weight.data *= mask_dict[name]
            print('module name is:', name, 'and weight size is:', module.weight.size())
            print('corresponding tensor is:', mask_dict[name].shape)
            module.weight.register_hook(lambda x: x*mask_dict[name])  #<-- problem here

#     model.linear1.weight.register_hook(lambda x: x*mask_dict['linear1'])
#     model.linear2.weight.register_hook(lambda x: x*mask_dict['linear2'])
#     model.linear3.weight.register_hook(lambda x: x*mask_dict['linear3'])

mask = l1norm_mask(model, 0.60)
apply_mask(model, mask)
mask.keys()

By printing some tensor shapes, it seems that they are all compatible for element-wise multiplication. The output is:

module name is: linear1 and weight size is: torch.Size([300, 784])
corresponding tensor is: torch.Size([300, 784])

module name is: linear2 and weight size is: torch.Size([100, 300])
corresponding tensor is: torch.Size([100, 300])

module name is: linear3 and weight size is: torch.Size([10, 100])
corresponding tensor is: torch.Size([10, 100])

dict_keys(['linear1', 'linear2', 'linear3'])

Trying to train the network gives the following error:

--------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-2e752da442fd> in <module>
----> 1 losses = [update(x,y,3e-3) for x,y in data.train_dl]

<ipython-input-18-2e752da442fd> in <listcomp>(.0)
----> 1 losses = [update(x,y,3e-3) for x,y in data.train_dl]

<ipython-input-17-ca0fbe546149> in update(x, y, lr)
      3     y_hat = model(x)
      4     loss = loss_fn(y_hat, y)
----> 5     loss.backward()
      6     opt.step()
      7     opt.zero_grad()

~/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    164                 products. Defaults to ``False``.
    165         """
--> 166         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    167
    168     def register_hook(self, hook):

~/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---> 99         allow_unreachable=True)  # allow_unreachable flag
    100
    101

<ipython-input-16-46b511a78892> in <lambda>(x)
     35             print('module name is:', name, 'and weight size is:', module.weight.size())
     36             print('correspoding tensor is:', mask_dict[name].shape)
---> 37             module.weight.register_hook(lambda x: x*mask_dict[name])
     38
     39 #     model.linear1.weight.register_hook(lambda x: x*mask_dict['linear1'])

RuntimeError: The size of tensor a (300) must match the size of tensor b (100) at non-singleton dimension 1

Update: I modified the code a bit and printed the the incoming gradient shape and shape of the mask I am passing in the loop. It seems that the way backward hooks are called, it always takes the first value ie ‘linear1’(=name) so same mask is getting passed to all the gradients resulting in mis-match matrix size error. Here’s the output I got:

shape of grad torch.Size([10, 100])  mask-shape torch.Size([10, 100])

shape of grad torch.Size([100, 300])  mask-shape torch.Size([10, 100])

shape of grad torch.Size([300, 784])  mask-shape torch.Size([10, 100])

shape of grad torch.Size([10, 100])  mask-shape torch.Size([10, 100])

shape of grad torch.Size([100, 300])  mask-shape torch.Size([10, 100])

shape of grad torch.Size([300, 784])  mask-shape torch.Size([10, 100])

and this repeats on…


title: How to implement a PyTorch NN from a directed graph

body:
I'm new to Pytorch and teaching myself, and I want to create ANNs that takes in a directed graph.  I also want to pass predefined weights & biases for each connection into it, but willing to ignore that for now.

My motivation for these conditions is that I'm trying to implement the NEAT algorithm, which is basically using a Genetic Algorithm to evolve the network.

For example, let
graph = dict{'1':[[], [4, 7]], '2':[[], [6]], '3':[[], [6]], '4':[[1, 7], []], '5':[[7], []], '6':[[2, 3], [7]], '7':[[1, 6], [4, 5]]}
represent the directed graph.

My code for what I'm thinking is:

class Net(torch.nn.Module):
    def __init__(self, graph):
        super(Net, self).__init__()
        self.graph = graph
        self.walk_graph()

    def walk_graph(self):
        graph_remaining = copy.deepcopy(self.graph)
        done = False  # Has every node/connection been processed?
        while not done:
            processed = []  # list of tuples, of a node and the nodes it outputs to
            for node_id in graph_remaining.keys():
                if len(graph_remaining[node_id][0]) == 0:  # if current node has no incoming connections
                    try:
                        # if current node has been processed, but waited for others to finish
                        if callable(getattr(self, 'layer{}'.format(node_id))):
                            D_in = len(eval('self.layer{}'.format(node_id)).in_features)
                            D_out = len(eval('self.layer{}'.format(node_id)).out_features)
                            setattr(self, 'layer{}'.format(node_id), torch.nn.Linear(D_in, D_out))
                        cat_list = [] # list of input tensors
                        for i in self.graph[node_id][0]: # search the entire graph for inputs
                            cat_list.append(globals()['out_{}'.format(i)]) # add incoming tensor to list
                        # create concatenated tensor for incoming tensors
                        # I'm not confident about this
                        globals()['in_{}'.format(node_id)] = torch.cat(cat_list, len(cat_list))
                    except AttributeError:  # if the current node hasn't been waiting
                        try:
                            setattr(self, 'layer{}'.format(node_id), torch.nn.Linear(len(self.graph[node_id][0]), len(self.graph[node_id][1])))
                        except ZeroDivisionError:  # Input/Output nodes have zero inputs/outputs in the graph
                            setattr(self, 'layer{}'.format(node_id), torch.nn.Linear(1, 1))
                    globals()['out_{}'.format(node_id)] = getattr(self, 'layer' + node_id)(globals()['in_{}'.format(node_id)])
                    processed.append((node_id, graph_remaining[node_id][1]))

            for node_id, out_list in processed:
                for out_id in out_list:
                    try:
                        graph_remaining[str(out_id)][0].remove(int(node_id))
                    except ValueError:
                        pass
                try:
                    del graph_remaining[node_id]
                except KeyError:
                    pass

            done = True
            for node_id in self.graph.keys():
                if len(graph_remaining[node_id][0]) != 0 or len(graph_remaining[node_id][1]) != 0:
                    done = False
        return None

I'm a little out of my comfort zone on this, but if you have a better idea, or can point out how this is fatally flawed, I'm all ears.  I know I'm missing a forward function, and could use some advice about how to restructure.


title: Feeding a 1D vector to Pytorch CNN

body:
How to feed a 1D vector to Pytorch CNN
Data is of the shape (244, 108). It contains percentage change of close value of a trade for each minute in one day ie 108 values and like that for 244 days. Basically its a 1D vector.

How to load this data to conv1d for regression. What will be the in_channel, out_channel and kernal_size?

Data:

x.shape = (243, 108)
y.shape = (243,)

Model (I tried):-

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.layer1 = torch.nn.Conv1d(in_channels=1, out_channels=1, kernel_size=3, stride=1)
        self.act1 = torch.nn.ReLU()
        self.act2 = torch.nn.AdaptiveMaxPool1d(1)
        self.layer2 = torch.nn.Conv1d(in_channels=1, out_channels=1, kernel_size=3, stride=1)
        self.act3 = torch.nn.ReLU()
        self.act4 = torch.nn.AdaptiveMaxPool1d(1)

        self.linear_layers = nn.Linear(1, 1)

    # Defining the forward pass
    def forward(self, x):
        x = self.layer1(x)
        x = self.act1(x)
        x = self.act2(x)
        x = self.layer2(x)
        x = self.act3(x)
        x = self.act4(x)
        x = x.reshape(x.shape[0], -1)
        x = self.linear_layers(x)
        return x


title: Do ReLU1 in PyTorch

body:
I want to use ReLU1 non-linear activation. ReLU1 is linear in [0,1] but clamps values less than 0 to 0 and clamps values more than 1 to 1.

It will be used only for the last layer of my deep net in PyTorch having a really high definition output of 2048x4096. Since the code has to be highly optimized in terms of speed and memory I do not know which of the following will be the best implementation.

Following are the two implementations I can think of for the tensor x:-

x.clamp_(min=0.0, max=1.0)
For this I am unable to see the source code given in its docs. So do not know if its the best choice. I will prefer in place operation since backpropagation can happen through it.

The second alternative I have is to use torch.nn.functional.hardtanh_(x, min_val=0.0, max_val=1.0). This is definitely a in place function and the source code says that it uses the C++ file torch._C._nn.hardtanh(input, min_val, max_val) so I think it will be fast.

Please suggest which is the most efficient implementation and another one if possible.

Thankyou


title: Problems initializing model in pytorch

body:
I can't initialize my model in pytorch and get:
TypeError                                 Traceback (most recent call last)
<ipython-input-82-9bfee30a439d> in <module>()
288 dataset = News_Dataset(true_path=args.true_news_file,
                            fake_path=args.fake_news_file,
289                         embeddings_path=args.embeddings_file)
 --> 290 classifier = News_classifier_resnet_based().cuda()
291 try:
292   classifier.load_state_dict(torch.load(args.model_state_file))

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in
__call__(self, *input, **kwargs)
548             result = self._slow_forward(*input, **kwargs)
549         else:
--> 550             result = self.forward(*input, **kwargs)
551         for hook in self._forward_hooks.values():
552             hook_result = hook(self, input, result)

TypeError: forward() missing 1 required positional argument: 'input'

Someone asked for code. It is given below

class News_classifier_resnet_based(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.activation = torch.nn.ReLU6()
        self.sigmoid = torch.nn.Sigmoid()

        self.positional_encodings = PositionalEncoder()

        self.resnet = list(torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True).children())

        self.to_appropriate_shape = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=77)

        self.conv1 = torch.nn.Conv2d(in_channels=1,out_channels=64,kernel_size=7,stride=2,padding=3)
        self.conv1.weight = torch.nn.Parameter(self.resnet[0].weight[:,0,:,:].data)
        self.center = torch.nn.Sequential(*self.resnet[1:-2])
        self.conv2 = torch.nn.Conv2d(in_channels=512, out_channels=1, kernel_size=1)
        self.conv3 = torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=7)

        self.title_conv = torch.nn.Sequential(
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=3,stride=3),
                    self.activation(),
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=2,stride=2),
                    self.activation(),
                    torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=2,stride=2)
                )
       self.title_lin = torch.nn.Linear(25,1)

       self.year_lin = torch.nn.Linear(10,1)
       self.month_lin = torch.nn.Linear(12,1)
       self.day_lin = torch.nn.Linear(31,1)
       self.date_lin = torch.nn.Linear(3,1)

       self.final_lin = torch.nn.Linear(3,1)

   def forward(self,x_in):
       #input shape - (batch_size, 3+title_len+seq_len, embedding_dim)
       #output shape - (batch_size, 1)
       year = x_in[:,0,:10]
       month = x_in[:,1,:12]
       day = x_in[:,2,:31]

       title = x_in[:,3:3+args.title_len,:]
       text = x_in[:,3+args.title_len:,:]
       title = self.positional_encodings(title)
       text = self.positional_encodings(text)

       text = text.unsqueeze(1)
       text = self.activation(self.to_appropriate_shape(text))
       text = self.activation(self.conv1(text))
       text = self.activation(self.center(text))
       text = self.activation(self.conv2(text))
       text = self.activation(self.conv3(text))
       text = text.reshape(args.batch_size,-1)

       title = title.unsqueeze(1)
       title = self.activation(self.title_conv(title))
       title = title.reshape(args.batch_size,-1)
       title = self.activation(self.title_lin(title))

       year = self.activation(self.year_lin(year))
       month = self.activation(self.month_lin(month))
       day = self.activation(self.day_lin(day))
       date = torch.cat([year,month,day], dim=-1)
       date = self.activation(self.date_lin(date))

       final = torch.cat([date,title,text], dim=-1)
       final = self.sigmoid(self.final_lin(final))

       return final

classifier = News_classifier_resnet_based().cuda()

What should I do? StackOverflow asked for more details. I'm trying to classify texts using word embeddings but problem lies in last line. I am working in google colab. Also when I created some models in other code blocks, I've got no problems


title: torch.random() TypeError: 'module' object is not callable

body:
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.random()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable
>>>

Do not know what's wrong with it.


title: Unable to access parameters of a model loaded with Lua

body:
I'm trying to run neural network-based code which load Vgg16 model using load fonction from the Torchfile package. The code uses this function from this file(https://github.com/zhanghang1989/PyTorch-Multi-Style-Transfer/blob/master/experiments/utils.py) to load the vgg16 model :
def init_vgg16(model_folder):
    """load the vgg16 model feature"""
    if not os.path.exists(os.path.join(model_folder, 'vgg16.weight')):
        if not os.path.exists(os.path.join(model_folder, 'vgg16.t7')):
            os.system(
                'wget http://cs.stanford.edu/people/jcjohns/fast-neural-style/models/vgg16.t7 -O ' + os.path.join(model_folder, 'vgg16.t7'))
        vgglua = load_lua(os.path.join(model_folder, 'vgg16.t7'))
        vgg = Vgg16()
        for (src, dst) in zip(vgglua.parameters()[0], vgg.parameters()):
            dst.data[:] = src
        torch.save(vgg.state_dict(), os.path.join(model_folder, 'vgg16.weight'))

which lead to the following error :
in init_vgg16
for (src, dst) in zip(vgglua.parameters()[0], vgg.parameters()):
TypeError: 'NoneType' object is not callable

I really do not know why I'm getting this error. My Torchfile package version in googleColab is 0.1.0 and torch is 0.3.1.
Any help please?


title: PyTorch: Target 1 is out of bounds

body:
I am new to Deep Learning and wondering how to modify my model to fix it.
It says Target 1 is out of bounds, so what parameter should I change to make it works. When the output is changed to 2, it works. However, the goal for the model is to predict 2 classes classification. Also, when output is 2, the training loss becomes nan.
The data is a dataframe with shape (15958, 4) transformed into tensor format.
Sorry Split_NN is a class:
# SplitNN
# to protect privacy and split
class SplitNN:
    def __init__(self, models, optimizers):
        self.models = models
        self.optimizers = optimizers

        self.data = []
        self.remote_tensors = []

    def forward(self, x):
        data = []
        remote_tensors = []

        data.append(self.models[0](x))

        if data[-1].location == self.models[1].location:
            remote_tensors.append(data[-1].detach().requires_grad_())
        else:
            remote_tensors.append(
                data[-1].detach().move(self.models[1].location).requires_grad_()
            )

        i = 1
        while i < (len(models) - 1):
            data.append(self.models[i](remote_tensors[-1]))

            if data[-1].location == self.models[i + 1].location:
                remote_tensors.append(data[-1].detach().requires_grad_())
            else:
                remote_tensors.append(
                    data[-1].detach().move(self.models[i + 1].location).requires_grad_()
                )

            i += 1

        data.append(self.models[i](remote_tensors[-1]))

        self.data = data
        self.remote_tensors = remote_tensors

        return data[-1]

    def backward(self):
        for i in range(len(models) - 2, -1, -1):
            if self.remote_tensors[i].location == self.data[i].location:
                grads = self.remote_tensors[i].grad.copy()
            else:
                grads = self.remote_tensors[i].grad.copy().move(self.data[i].location)

            self.data[i].backward(grads)

    def zero_grads(self):
        for opt in self.optimizers:
            opt.zero_grad()

    def step(self):
        for opt in self.optimizers:
            opt.step()

Below are the codes:
Model set up: The Model is a sequential deep learning model, which I tried to use nn.linear to generated binary prediction.
torch.manual_seed(0)

# Define our model segments

input_size = 3
hidden_sizes = [128, 640]
output_size = 1

# original models
models = [
    nn.Sequential(
        nn.Linear(input_size, hidden_sizes[0]),
        nn.ReLU(),
        nn.Linear(hidden_sizes[0], hidden_sizes[1]),
        nn.ReLU(),
    ),
    nn.Sequential(nn.Linear(hidden_sizes[1], output_size), nn.LogSoftmax(dim=1)),
]

# Create optimisers for each segment and link to them

optimizers = [
    optim.SGD(model.parameters(), lr=0.03,)
    for model in models
]

Train model is here:
def train(x, target, splitNN):

    #1) Zero our grads
    splitNN.zero_grads()

    #2) Make a prediction
    pred = splitNN.forward(x)

    #3) Figure out how much we missed by
    criterion = nn.NLLLoss()
    loss = criterion(pred, target)

    #4) Backprop the loss on the end layer
    loss.backward()

    #5) Feed Gradients backward through the nework
    splitNN.backward()

    #6) Change the weights
    splitNN.step()

    return loss, pred

Finally the training part, also the part where problem happen:
the send function is for assigning model to the nodes, cuz this is set up to simulating federated learning.
for i in range(epochs):
    running_loss = 0
    correct_preds = 0
    total_preds = 0

    for (data, ids1), (labels, ids2) in dataloader:
        # Train a model
        data = data.send(models[0].location)
        data = data.view(data.shape[0], -1)
        labels = labels.send(models[-1].location)

        # Call model
        loss, preds = train(data.float(), labels, splitNN)

        # Collect statistics
        running_loss += loss.get()
        correct_preds += preds.max(1)[1].eq(labels).sum().get().item()
        total_preds += preds.get().size(0)

    print(f"Epoch {i} - Training loss: {running_loss/len(dataloader):.3f} - Accuracy: {100*correct_preds/total_preds:.3f}")

The error show the problem occurs at loss, preds = train(data.float(), labels, splitNN)
The actual error message:
During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target 1 is out of bounds.

Please help me. Thank you


title: pytorch - GroupNorm with momentum

body:
After looking at the documentation of pytorch I couldn't find the module of GroupNorm with momentum, there is only an implementation which doesn't use it (which is useless to me since I would want to use AdaBN, or AdaGN I should say).

Therefore my idea was to take the BatchNorm class and extend it with num_groups and override the method that actually calculate mean and variance by firstly reshaping the input and grouping channels together, so to emulate the GroupNorm behavior.

In order to do this I looked into the source code until I found a dead end in torch.nn.functional.batch_norm. That function internally calls another function torch.batch_norm(...) but I couldn't find the source code for it.

Do you know where to look for that part? Or do you think there is a better way to implement this? I tried looking online for other's works, but I only found unfinished ones.


title: How to run network with multiple independent inputs in parallel in Pytorch?

body:
I have a neural network with two separate vectors as inputs, similar to this question.
Both inputs are encoded and then processed further. But until then, the encoding of those inputs is completely independent.
How can I parallelize the encoding phase in pytorch?
A minimal example of my code:

class MyModel(nn.Module):
    def __init__(params):
        super(MyModel self).__init__()

        self.encoder1 = Encoder1()
        self.encoder2 = Encoder2()

    def forward(x1, x2):
        # how to calculate both encodings in parallel?
        enc1 = self.encoder1(x1)
        enc2 = self.encoder2(x2)
        return some_func(enc1, enc2)

I checked pytorch forums and learned a bit about DataParallel, but i am not sure how to fit it to my case and on only 1 GPU.


title: How does PyTorch conv1D handle the padding for variable length sequences?

body:
I have a batch of variable length sequences padded with 0 (padding = post) and would like to perform conv1D to generate feature maps. How does PyTorch conv1D handle padded sequences? For example, the rnn (recurrent neural network) and crf (conditional random fields) layers do allow masking the padded sequences, but I did not see such an option in conv1D.

I did check out the conv1D layer and there is an option to pad the sequences, but it does not seem to be similar to the padding and masking I am talking about.

m = nn.Conv1d(100, 6, 2, padding= 1)

I will be really grateful for any help.


title: Pytorch validation accuracy does not improve, while train accuracy improve

body:
I used an existing model (vgg-16), and trained the last layers of the model.]
age_list = ['(0, 3)', '(4, 7)', '(8, 14)', '(15, 24)', '(25, 37)', '(38, 47)', '(48, 59)', '(60, 100)']
net = torch.hub.load('pytorch/vision:v0.6.0', 'vgg16_bn', pretrained=True)
data_dir=""
model_name="vgg"
num_classes=len(age_list)
feature_extract=True

def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False
def initialize_model(model_name, num_classes, feature_extract, use_pretrained=True):
    # Initialize these variables which will be set in this if statement. Each of these
    #   variables is model specific.
    model_ft = None
    input_size = 0
    model_ft = net
    set_parameter_requires_grad(model_ft, feature_extract)
    num_ftrs = model_ft.classifier[6].in_features
    model_ft.classifier = nn.Sequential(
            nn.Linear(25088, 4096,bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(p=0.5,inplace=False),
            nn.Linear(4096,4096,bias=True),
            nn.ReLU(inplace=True) ,
            nn.Dropout(p=0.5,inplace=False),
            nn.Linear(4096,8,bias=True),

        )
    input_size = 224
    return model_ft, input_size

My loss function and optimizer
    criterion =nn.CrossEntropyLoss()
    optimizer = optim.Adam(model_ft.parameters(), lr=0.0001)

I have 2 dataset of faces, one for the train (9781 images) and one for the validation (40508 images).
The dataset is variety and contains all ages and genders.
By the end of the training, this was the best model :
    Best val Acc: 0.331070

These are the graphs that show the accuracy and loss of each epoch:

I'd like to know where I can improve my model to get better results.
Thanks a lot.


title: pytorch beginner :torch.data.new() torch.new()

body:
I have a small question, what is the difference between tensor.data.new() and tensor.new()? It seems they all return an empty tensor with the same dtype and device as the self tensor.
Thank you


title: PyTorch LSTM categorical model - output to target mapping

body:
I have a network which outputs a vector of length two. My targets are in the form of 1 or zeros, referring to two possible categories. What is the best way to get the loss - i.e. should I transform the targets, for example into a dimension 2 vector, or should I transform the output of the network, e.g. take the location of the max number as the output?
My network looks like:
class LSTMClassifier(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm1 = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.lstm2 = nn.LSTM(hidden_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc1 = nn.Linear(hidden_dim, 32)
        self.fc2 = nn.Linear(32, 1)
        self.dropout = nn.Dropout(p=0.2)
        self.batch_normalisation1 = nn.BatchNorm1d(layer_dim)
        self.batch_normalisation2 = nn.BatchNorm1d(2)
        self.activation = nn.Softmax(dim=2)

    def forward(self, x):
        h0, c0 = self.init_hidden(x)
        out, (hn1, cn1) = self.lstm1(x, (h0, c0))
        out = self.dropout(out,)
        out = self.batch_normalisation1(out)

        h1, c1 = self.init_hidden(out)
        out, (hn2, cn2) = self.lstm2(out, (h1, c1))
        out = self.dropout(out)
        out = self.batch_normalisation1(out)

        h2, c2 = self.init_hidden(out)
        out, (hn3, cn3) = self.lstm2(out, (h2, c2))
        out = self.dropout(out)
        out = self.batch_normalisation1(out)

        out = self.fc1(out[:, -1, :])
        out = self.dropout(out)
        out = self.fc2(out)
        return out

    def init_hidden(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        return [t for t in (h0, c0)]

    def pred(self, x):
        out = self(x)
        return out > 0

An example of input to this network is:
tensor([[[0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],
         [2.3597e-04, 1.1507e-02, 8.7719e-02, 6.1093e-02, 9.5556e-01],
         [2.1474e-03, 5.3805e-03, 9.6491e-02, 2.2508e-01, 8.2222e-01]]])

which has shape torch.Size([1, 3, 5]). The target is currently 1 or 0. However, the network outputs a vector such as:
tensor([[0.5293, 0.4707]], grad_fn=<SoftmaxBackward>)

What would be the best way to set up the loss between these target and the network output?
Update:
I can now train the model as suggested in the answers as:
model = LSTMClassifier(5, 128, 3, 1)
Epochs = 10
batch_size = 32

criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=1e-6)

for epoch in range(Epochs):
    if epoch == 0:
        accurate = 0
        for X_instance, y_instance in zip(val_x, val_y):
            if int(y_instance) == 1 and model.pred(X_instance.view(-1, 3, 5)).item():
                accurate += 1
        print(f"Untrained accuracy test set: {accurate/len(val_x)}")
    print(f"Epoch {epoch + 1}")

    for n, (X, y) in enumerate(train_batches):
        model.train()
        optimizer.zero_grad()
        y_pred = model(X)
        loss = criterion(y_pred, y)
        loss.backward()
        optimizer.step()

    model.eval()
    accurate = 0
    for X_instance, y_instance in zip(val_x, val_y):
        if int(y_instance) == 1 and model.pred(X_instance.view(-1, 3, 5)).item():
            accurate += 1
    print(f"Accuracy test set: {accurate/len(val_x)}")


title: Split DataLoader PyTorch

body:
Is it possible to split a dataloader object of training dataset into training and validation dataloader?
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

train_dataset = datasets.ImageFolder(train_data_directory, transform=transforms.ToTensor())
# Data loader
train_loader = DataLoader(train_dataset, batch_size=100, shuffle=True)

Now I would like to split a train_loader to train and validation dataloader.