Stochastic Processes
A stochastic process \(X(t), t \in T\) is a collection of random variables. For each \(t \in T\), \(X(t)\) is a random variable. The index \(t\) is often interpreted as time, and as a result, we refer to \(X(t)\) as the state of the process at time \(t\).
The set \(T\) is called the index set of the process. When \(T\) is a countable set, the stochastic is a discretetime process. If \(T\) is an interval of the real line, the process is said to be a continuoustime process.
Focus: Discrete time, discrete state space Markov Chain

Stochastic = random

A stochastic process describes random phenomena that change over time

values that \(X_t\)‘s take

set of all possible states, denoted by \(\mathcal{S}\).

can be thought of as time. If \(T = \{0, 1, 2, \dots \}\) then it is a discretetime process. If \(T\) is an interval, it is a continuous time process.
Each \(X_t\) is a random variable.
Example of stochastic process: Gambler’s ruin
 A gambler starts with an initial fortune of \(k\) dollars.
 The gambler plays against \(B\) with an initial fortune of \(Nk\) dollars.
 Each game he bets $1, wins with probability \(p\)
 Let \(\{X_t = t = 0,1,2 \dots\}\) represent his fortune as the betting goes on.
 Game only stops when either gambler or \(B\) is ruined.
 Here \(\mathcal{S} = \{0,1,\dots,N\}\)
 For a realization of the results of the first 10 games, (here \(p=1/2\)):
sample(c(1, 1), 10, replace=T)
Reference Textbooks
, Ross, n.d., @pinsky2010introduction