#001 Linear Algebra: Vector addition and scalar-vector multiplication
Highlights: In this post we are going to talk about vectors. They are the fundamental building blocks in Linear Algebra. We will give an intuitive definition what the vectors are, where we use them, how we add them and multiply with scalars. We provide a code examples to demonstrate how to work with vectors in Python.
Tutorial Overview:
- What is a vector?
- Examples of vectors in Linear Algebra
- Definition of a vector
- Vector addition
- Scalar-vector multiplication
1. What is a vector?
So, what exactly is a vector? Let’s talk about this.
One simple way to think about a vector is to imagine it as an arrow that points in space. If we recall from elementary physics, a vector has a direction where its points. In addition, it also has a length.
For instance, a force can be illustrated and defined by means of vectors. The longer the arrow, the stronger is the force that we are applying. Again, in physics, we can move our vector in space, but as long as direction and length are the same, it is going to be the same vector.
Most commonly, we encounter vectors that sit in a two-dimensional plane. For those vectors we also say that they are two-dimensional.
Vectors can also live in a 3D space. Those vectors are called three-dimensional vectors. They are defined with three coordinates (x, y, z).
A 3D vector is often encountered in physics. In the image below we have a random, and the so called Brownian motion. Every molecule has a location and a velocity that is determined by a vector direction and length (intensity).
2. Vector examples
On the other hand, in machine learning, we can define vectors as an arbitrary ordered set of numbers. For instance, when we work with regression problems, we can define a vector that describes a second-hand car. For instance, we want to develop a smart AI agent for finding a car in a huge database that we can call “a good buy”. In such a case, a car can be described with its price, mileage, engine power and a year of the production. Here, the most important thing is that every “car vector” is described with these attributes in the same order. So, in this example, we have a vector of dimension (size) 4.
There are other examples, where the size of the vectors can be relatively large. For instance, when working with DNAs, scientists process vectors of gene expressions whose size is at the order of thousands of elements.
In addition, image pixels can be represented as an RGB vector (red, green, blue). For instance, we can define the following vectors that we use for the color image processing.
3. Definition of a vector
Within the scope of linear algebra, a vector is defined under the operation of summation and the multiplication by a scalar.
In addition, when we work with vectors in linear algebra we define them as “arrows” whose tail is at the origin of the coordinate system.
Let’s focus our attention on two dimensions for the moment. We have a horizontal line (x-axis), and a vertical line (y-axis). The place where they intersect is called the origin, which we should think of as the center of space and the root of all vectors. After choosing an arbitrary length to represent 1, we make tick-marks on each axis to represent the distance.
The coordinates of a vector is a pair of numbers that basically give instructions how to get from the origin of that vector (the tail) to its tip . The first number, painted in green, tells you how far to move along the \(x \)-axis. The second number painted in red tells you how far to move parallel to the \(y \)-axis after that.
What about in three dimensions? Here, we have 3 coordinates. We add a third axis, called the z-axis, which is perpendicular to both the \(x \) and \(y \)-axes. In this case, each vector is associated with an ordered triplet of numbers: the first tells you how far to move along the \(x \)-axis, the second tells you how far to move parallel to the \(y \)-axis, and the third one tells you how far to then move parallel to this new \(z \)-axis.
Every topic in linear algebra is going to center around two operations: vector addition and multiplication by scalars. Luckily, each one is pretty straightforward to define.
Definition of a vector
A vector is an ordered finite list of numbers. Vectors are usually written as vertical arrays, surrounded by square or curved brackets, as in
$$ \begin{bmatrix}-1.1\\0.0\\3.6\\-7.2\end{bmatrix} $$
$$ \begin{pmatrix}-1.1\\0.0\\3.6\\-7.2\end{pmatrix} $$
They can also be written as numbers separated by commas and surrounded by parentheses. In this notation style, the vector above is written as
$$ \left ( -1.1, 0.0, 3.6, -7.2 \right ) $$
The elements (or entries, coefficients, components) of a vector are the values in the array. The size (also called dimension or length) of the vector is the number of elements it contains.
A vector of size \(n \) is called an \(n \) -vector.
Two vectors \(a \) and \(b \) are equal, which we denote \(a = b \), if they have the same size, and each of the corresponding entries are the same. If \(a \) and \(b \) are \(n \) -vectors, then \(a = b \) means \(a_{1}= b_{1} \), . . . , \(a_{n}= b_{n} \).
The numbers or values of the elements in a vector are called scalars.
Indexing
We should give a couple of warnings concerning the subscripted index notation \(a_{i} \). The first warning concerns the range of the index. In Python, arrays of length \(n \) are indexed from \(i = 0 \) to \(i = n – 1 \).
Zero vectors
A zero vector is a vector with all elements equal to zero. Sometimes the zero vector of size \(n \) is written as \(0_{n} \), where the subscript denotes the size. But usually a zero vector is denoted just \(0 \), the same symbol used to denote the number \(0 \).
Unit vectors
A (standard) unit vector is a vector with all elements equal to zero, except one element which is equal to one. The \(i \) – th unit vector (of size \(n \)) is the unit vector with \(i \) – th element one, and denoted \(e_{i} \). For example, the vectors
\( e_{1}= \begin{bmatrix}1\\0\\0\end{bmatrix} e_{2}= \begin{bmatrix}0\\1\\0\end{bmatrix} e_{3}= \begin{bmatrix}0\\0\\1\end{bmatrix} \)
are the three-unit vectors of size \(3 \).
Ones vector
We use the notation \(1_{n} \) for the \(n \) -vector with all its elements equal to one. We also write \(1 \) if the size of the vector can be determined from the context.
Position vs. vector
The \(2 \) -vector \(v \) specifies the position (shown as a dot) with coordinates \(x \) and \(y \) in a plane.
The \(2 \) -vector x represents a displacement in the plane (shown as an arrow) by \(x \) in the first axis and \(y \) in the second.
4. Vector addition
Let’s say that we have two vectors, one pointing up, and a little to the right (\(\vec{v}\)), and the other one pointing right, and down a bit (\(\vec{w}\)). To add these two vectors, we move the second one so that its tail sits at the tip of the first one; then if you draw a new vector from the tail of the first one to where the tip of the second one now sits, that new vector is their sum.
You may think why is this a reasonable thing to do? Well, the way we think about it is that each vector represents a certain movement-a step with a certain distance and direction in space. If you take a step along the first vector, then take a step in the direction and distance described by the second vector, the overall effect is just the same as if you moved along the sum of those two vectors to start with.
Here is an illustration how two vectors are added.
Two vectors of the same size can be added together by adding the corresponding elements, to form another vector of the same size, called the sum of the vectors. Vector addition is denoted by the symbol +.
$$ \begin{bmatrix}0\\7\\3\end{bmatrix}+\begin{bmatrix}1\\2\\0\end{bmatrix}= \begin{bmatrix}1\\9\\3\end{bmatrix} $$
Vector subtraction is similar. As an example,
$$ \begin{bmatrix}1\\9\end{bmatrix}-\begin{bmatrix}1\\1\end{bmatrix}= \begin{bmatrix}0\\8\end{bmatrix} $$
Properties
Several properties of vector addition are easily verified. For any vectors \(a, b \), and \(c \) of the same size we have the following.
• Vector addition is commutative: \(a + b = b + a \).
• Vector addition is associative: \((a + b) + c = a + (b + c) \). We can therefore write both as \(a + b + c \).
• \(a + 0 = 0 + a = a \). Adding the zero vector to a vector has no effect. (This is an example where the size of the zero vector follows from the context: It must be the same as the size of \(a \).)
• \(a − a = 0 \). Subtracting a vector from itself yields the zero vector. (Here too the size of \(0 \) is the size of \(a \).)
5. Scalar-vector multiplication
Another operation is scalar multiplication or scalar-vector multiplication, in which a vector is multiplied by a scalar (i.e., number), which is done by multiplying every element of the vector by the scalar. Scalar multiplication is denoted by juxtaposition, typically with the scalar on the left, as in
$$ \left ( -2 \right )\cdot \begin{bmatrix}1\\9\\6\end{bmatrix}= \begin{bmatrix}-2\\-18\\-12\end{bmatrix} $$
Scalar-vector multiplication can also be written with the scalar on the right, as in
$$ \begin{bmatrix}1\\9\\6\end{bmatrix}\cdot \left ( 1.5 \right ) = \begin{bmatrix}1.5\\13.5\\9\end{bmatrix} $$
This process of stretching the direction of a vector is called scaling, and whenever you catch a number like 2 or 1/3 or -1.8 acting like this (scaling some vector) you call it a scalar. In fact, throughout linear algebra, one of the main things that numbers do is scale vectors, so it’s common to use the word “scalar” pretty much interchangeably with the word “number”.
Summary
Linear algebra gives the data analyst a nice way to conceptualize many lists of numbers in a visual way, which can seriously clarify patterns in data, and give a global view of what certain operations do.
In the following posts you will see what we mean when we say that linear algebra topics tend to center around these two fundamental operations: vector addition and scalar-vector multiplication. In the next post we will start getting into some pretty neat concepts surrounding vectors, like span, basis and linear dependence.
References:
- Vectors, what even are they? | Essence of linear algebra, chapter 1. 3Blue1Brown.
