# Notes on PyTorch Tensor Data Types

## Contents

In PyTorch,
`Tensor`

is the
primary object that we deal with (`Variable`

is just a thin wrapper class for
`Tensor`

). In this post, I will give a summary of pitfalls that we should avoid
when using Tensors. Since `FloatTensor`

and `LongTensor`

are the most popular
`Tensor`

types in PyTorch, I will focus on these two data types.

# Tensor operations

## Tensor and Tensor operation

For operations between Tensors, the rule is strict. Both Tensors of the operation must have the same data type, or you will see error messages like

TypeError: sub received an invalid combination of arguments - got (float), but expected one of:

* (int value)

didn’t match because some of the arguments have invalid types: (!float!) * (torch.LongTensor other)

didn’t match because some of the arguments have invalid types: (!float!) * (int value, torch.LongTensor other)

As another example, several loss functions like
`CrossEntropyLoss`

require that the target should be torch `LongTensor`

. So before doing
operations, make sure that your input `Tensor`

types match the function
definitions.

It is easy to convert the type of one `Tensor`

to another `Tensor`

. Suppose `x`

and `y`

are `Tensor`

of different types. You can use `x.type(y.type())`

or
`x.type_as(y)`

to convert `x`

to the type of `y`

.

## Tensor and scalar operation

For `FloatTensor`

, you can do math operations (multiplication, addition,
division etc.) with a scalar of type `int`

or `float`

. But for `LongTensor`

,
you can only do math operation with `int`

scalar **but not float**.

# Why do some losses require target to be LongTensor?

According to PyTorch
developers,
some use cases requires that the target be `LongTensor`

type and int just can
not hold the target value.

# FloatTensor or DoubleTensor

For deep learning, precision is not a very important issue. Plus, GPU can not
process double precision very well. So `FloatTensor`

is enough, which is also
the default type for model parameters.

# NumPy array and torch Tensor

## Shared memory or not?

You can use `torch.from_numpy()`

method to convert a NumPy array to
corresponding torch `Tensor`

, which will share underlying memory with NumPy
array. To convert `Tensor`

`x`

to NumPy array, use `x.numpy()`

to convert it to
a NumPy array, which also shares the memory with original `Tensor`

.

Does torch `Tensor`

and Numpy array always share the underlying memory? The
short answer is no. If their underlying data type is not compatible, a copy of
original data will be made. For example, if you try to save torch `FloatTensor`

as numpy array of type `np.float64`

, it will trigger a deep copy.

## Correpsondece between NumPy and torch data type

It should be noted that not all NumPy arrays can be converted to torch
`Tensor`

. Below is a table showing NumPy data types which is convertable to
torch `Tensor`

type.

NumPy data type | Tensor data type |
---|---|

`numpy.uint8` |
`torch.ByteTensor` |

`numpy.int16` |
`torch.ShortTensor` |

`numpy.int32` |
`torch.IntTensor` |

`numpy.int` |
`torch.LongTensor` |

`numpy.int64` |
`torch.LongTensor` |

`numpy.float32` |
`torch.FloatTensor` |

`numpy.float` |
`torch.DoubleTensor` |

`numpy.float64` |
`torch.DoubleTensor` |

## Speed comparison between NumPy and torch operations

I am curious to know the speed difference between torch Tensor operation and
equivalent NumPy ndarray operations. I do it in Jupyter-console using the
builtin magic `%timeit`

.

```
import torch
import numpy as np
# torch Tensor on CPU
x = torch.rand(1, 64)
y = torch.rand(5000, 64)
%timeit z=(x*y).sum(dim=1)
# torch Tensor on GPU
x, y = x.cuda(), y.cuda()
%timeit z = (x*y).sum(dim=1)
# numpy ndarray on CPU
x = np.random.random((1, 64))
y = np.random.random((5000, 64))
%timeit z = (x*y).sum(axis=1)
```

The result is listed on the following table:

Data type and device | Average operation time |
---|---|

Tensor on CPU | 938 $\mu s$ |

Tensor on GPU | 38.9 $\mu s$ |

NumPy ndarray (on CPU) | 623 $\mu s$ |

It is pretty clear that Tensor operations on GPU runs orders of magnitute faster than operations on CPU. NumPy, due to its excellent implementation of its core in C, runs a little bit faster than Tensor on CPU.

# Convert scalar to torch Tensor

You can convert a scalar to `Tensor`

by providing the scalr to the `Tensor`

constructor, which will not achieve what you want. For
example,`torch.Tensor(1)`

will not give you a `Tensor`

which contains float 1.
Instead, the produced `Tensor`

is something like

1.00000e-20 * 5.4514

[torch.FloatTensor of size 1]

To achieve what you want, you have to provide a list with single element 1 to
the `Tensor`

constructor.