datahacker.rs@gmail.com

#002 PyTorch – The main data structure of PyTorch?

#002 PyTorch – The main data structure of PyTorch?

Highlights: Welcome everyone! In this post we will cover the main data structure of PyTorchtensors. However, let’s first get a quick overview of what PyTorch is and why it has been popular lately. Next, in the folowing posts we will use this knowledge to build interesting applications.

Tutorial Overview:

Before we go over the explanation, you can download code from our GitHub repo.

1. What is PyTorch?

PyTorch is an open-source Python framework released from the Facebook AI Research Team. Its main purpose is for the developing of deep learning models. It’s a Python-based scientific computing package with the main goal to:

• Have characteristic of a NumPy library to harness the power of GPUs but with stronger acceleration.
• In addition, it provides remarkable flexibility and speed during the implementation and the development of deep neural network architectures.

PyTorch Community

PyTorch community is growing in numbers on a daily basis. It has been largely used by many scientific researchers. With its rising popularity, more people are adopting PyTorch within their research labs to develop sophisticated deep learning models.

If you take a look at the GitHub repository, there are 47, 490 contributors working on improvement to the existing PyTorch functionalities.

Currently, it has been applied by Tech giants such as Facebook, Tesla, Uber, Nvidia.

In a research community fields it has been mainly used for neural network/deep learning, computer vision, image recognition, and NLP. After the release of the version 1.0, it has helped researchers to tackle four major difficulties such as:

• Large-scale reworking
• Reducing training time
• Python Programming language stability.

Now, let’s see how we can create tensors. We will show how to do simple operations with them, and also we will show the interact link between tensors and NumPy arrays.

2. What are Tensors?

In linear algebra tensor is a generalization of vectors and matrices. For instance a tensor with only one dimension is a vector. In addition, matrices are tensors with two dimensions. Then, we can also have a three dimension tensor. Example of a three dimension tensor is image represented in a computer with three channels – red, green and blue (RGB image).This can continue to expand to four dimensional tensors and so on. However, we will mainly work with the tensors up to dimension of three.

First, let’s import our torch library to create a 1d, 2d, and 3d tensors. By default, these tensors are of float type which is recommended and will be used throughout these posts.

Creating tensors

# Import necessary library
import torch

vector_data = [20., 40., 60., 80., 100.]
vector = torch.tensor(vector_data)
print(vector, end='\n\n')

# Create a 2D Tensor of size 2x3
Matrix_data = [[20., 40., 60.], [80., 100., 120.]]
Matrix = torch.tensor(Matrix_data)
print(Matrix, end='\n\n')

# Create a 3D tensor of size 3x2x2.
Tensor_data = [[[20., 40.], [60., 80.]],
[[100., 120.], [140., 160.]],
[[180., 200.], [220., 240.]]]
Tensor = torch.tensor(Tensor_data)
print(Tensor, end='\n\n')
Output:
tensor([ 20., 40., 60., 80., 100.])

tensor([[ 20., 40., 60.], [ 80., 100., 120.]])

tensor([[[ 20., 40.], [ 60., 80.]], [[100., 120.], [140., 160.]], [[180., 200.], [220., 240.]]])

Within PyTorch we cal also perform indexing. Let’s see how we can access individual elements within a vector, matrix and tensor.

Indexing Using PyTorch

# Indexing into Vector and get a scaler
print(Vector[0])

# Get a Python number from it
print(vector[0].item())

# Indexing into Matrix and get a vector
print(Matrix[0])

# Indexing into Tensor and get a matrix
print(Tensor[0])
Output:
tensor(20.)
20.0
tensor([20., 40., 60.])
tensor([[20., 40.], [60., 80.]])

In addition we can we a tensor with random numbers. In particular we can use these numbers which are sampled with a normal distribution (Gaussian distribution). To accomplish this we will use a function torch.randn().

Generating Data

z = torch.randn((4, 4))
print(z)
Output: tensor([[ 1.2589, 0.3926, 1.0688, -2.7936],
[-2.0852, -0.5320, 0.5999, -0.5298],
[ 0.2900, -0.3347, 1.0306, -0.9973],
[-1.9540, -1.1123, -0.3074, 1.4458]])

Intuitively, we can also perform basic mathematical operations on tensors in various ways. Let’s take a look at some examples such as an addition, subtraction, multiplication and division.

Operations with Tensors

x = torch.tensor([4., 2., 6.])
y = torch.tensor([3., 19., 12.])

torch.add(x, y)
Output: tensor([ 7., 21., 18.])
# Subtraction
torch.sub(x, y)
Output: tensor([ 1., -17., -6.])
# Multiplication
torch.mul(x, y)
Output: tensor([12., 38., 72.])
# Division
torch.div(x, y)
Output: tensor([1.3333, 0.1053, 0.5000])

We will proceed by showing how to concatenate tensors. This can be done in a row-wise or a column-wise manner. Note that if we have tensors of different sizes, PyTorch will raise an exception. For this choice between a row-wise and a column-wise in a function torch.cat()we use an argument axis=0 or axis=1.

# Concatenate columns: Row-wise, axis=0 is by default.
a_1 = torch.randn(2, 2)
b_1 = torch.randn(2, 2)
c_1 = torch.cat([a_1, b_1], axis=0)
print(c_1)
Output:
tensor([[-0.5220, 0.1275],
[ 0.5694, 1.6213],
[-0.5436, -1.7412],
[-0.2146, -1.7271]])
# Concatenate columns: Column-wise, axis=1.
a_2 = torch.randn(2, 3)
b_2 = torch.randn(2, 1)

c_2 = torch.cat([a_2, b_2], axis=1)
print(c_2)
Output:
tensor([[ 0.2604, -0.8682, 0.9397, 0.5422],
[ 0.0544, -0.9327, -1.3670, 0.2649]])
# If we have tensors of different shape, torch will raise an exception.
torch.cat([a_2, b_2])
Output:
RuntimeError: Sizes of tensors must match except in dimension 0. Got 3 and 1 in dimension 1

How may we overcome this unpleasant event? Well, with the use of the .view() method PyTorch offers, we can reshape our tensor. This is very crucial because many neural networks expect their input to be of a certain shape and, therefore, support for a data reshape is necessary.

Reshaping Tensors

x = torch.randn(3, 1, 2)
print(x)
Output:
tensor([[[ 0.4054, 0.6458]],
[[-0.3763, -0.7082]],
[[-0.7617, 0.0891]]])
print(x.view(3, 2))  # Reshape to 3 rows, 2 columns
Output:
tensor([[ 0.4054, 0.6458],
[-0.3763, -0.7082],
[-0.7617, 0.0891]])
# Same as above.  If one of the dimensions is -1, its size can be deduced
print(x.view(2, -1))
Output:
tensor([[ 0.4054, 0.6458, -0.3763],
[-0.7082, -0.7617, 0.0891]])

Numpy to Torch and back

PyTorch is very flexible and allows us to do conversion between Numpy arrays and Torch tensors. We can create a tensor from a Numpy array using the command torch.from_numpy(). In addition, we can go back from a tensor into a NumPy array using .numpy() function.

import numpy as np # Numerical Computation Library

X = np.random.randn(4, 4) # Generate random numbers
print(X)
print(type(X)) # A numpy array
Output:
[[ 0.29461242 0.74876946 1.48368907 -0.04842474]      [-2.04944721 -2.07051625 0.93955147 -1.20813359]
[ 0.92787283 -1.63746277 -1.56531153 -0.85409993] [-0.47695945 0.14266573 -1.41436873 0.46275093]]
<class 'numpy.ndarray'>
y = torch.from_numpy(x) # Create a tensor from numpy array
print(y)
print(type(y)) # A tensor
Output:
tensor([[ 0.2946, 0.7488, 1.4837, -0.0484],
[-2.0494, -2.0705, 0.9396, -1.2081],
[ 0.9279, -1.6375, -1.5653, -0.8541],
[-0.4770, 0.1427, -1.4144, 0.4628]],
dtype=torch.float64)
<class 'torch.Tensor'>
print(y.numpy()) # Convert the tensor into Numpy array
print(type(y.numpy()))
Output:
[[ 0.29461242 0.74876946 1.48368907 -0.04842474]
[-2.04944721 -2.07051625 0.93955147 -1.20813359]
[ 0.92787283 -1.63746277 -1.56531153 -0.85409993]
[-0.47695945 0.14266573 -1.41436873 0.46275093]]
<class 'numpy.ndarray'>

PyTorch includes a module called AutoGrad that calculates the gradients of function which is written by using a tensor. This is helpful because it keeps an eye on operations performed on a tensor. When we execute the .backwards() method on a tensor, it goes through all operations performed and calculates the gradients with respect to the input parameters.

Note: We need to tell PyTorch that we want to use AutoGrad on a specific tensor. Let’s take an example.

We create a Variable, then give it requires_grad = true. This tells PyTorch to keep track of the operations on this tensor and then calculate the gradient. In case we want to get the gradient, then it will calculate it for us. Note: If you create a tensor and don’t want to calculate the gradient for it, set requires_grad=False.

Note: To turn off gradients all together globally, we used the command torch.set_grad_enabled(True|False).

Computation Graphs and Automatic Differentiation

from torch.autograd import Variable

y = (6 * x) ** 2
# The last operation performed
print(y.grad_fn) 
Output:
<PowBackward0 object at 0x7f6af5454940>

Printing out our variable x.grad returns None. Since we have taken only the forward pass and haven’t calculated the gradients yet.

print(x.grad)
Output: None

The gradients are computed with respect to our variable $$y$$ with y.backward(). This does a backward pass through the operations that created $$y$$.

$$f^{\prime}(x)=\frac{\partial}{\partial x} 6 x^{2}=(2 \times 6) x^{2-1}=12 x=12 \times 2=144$$

By taking the derivative of $$y$$ with respect to $$x$$ we get the original value.

y.backward()
print(x.grad)
Output:
tensor([144.])

Summary

In conclusion, we have gone through the PyTorch data structures. We have learned who developed it and the reasons why it was designed. We also explored what tensors are, where we can use them, and we showed some basic operations with them. We explained what is Autograd package which provides automatic differentiation for all operations on tensors. In the next post, we will see how to implement a single layer neural network and make in performing binary classification.

Interactive Colab notebook can be found at the following link Run in Google Colab.