Jethro's Braindump

Stochastic Processes

A stochastic process \(X(t), t \in T\) is a collection of random variables. For each \(t \in T\), \(X(t)\) is a random variable. The index \(t\) is often interpreted as time, and as a result, we refer to \(X(t)\) as the state of the process at time \(t\).

The set \(T\) is called the index set of the process. When \(T\) is a countable set, the stochastic is a discrete-time process. If \(T\) is an interval of the real line, the process is said to be a continuous-time process.

Focus: Discrete time, discrete state space Markov Chain

  • Stochastic = random

  • A stochastic process describes random phenomena that change over time

  • values that \(X_t\)‘s take

  • set of all possible states, denoted by \(\mathcal{S}\).

  • can be thought of as time. If \(T = \{0, 1, 2, \dots \}\) then it is a discrete-time process. If \(T\) is an interval, it is a continuous time process.

Each \(X_t\) is a random variable.

Example of stochastic process: Gambler’s ruin

  1. A gambler starts with an initial fortune of \(k\) dollars.
  2. The gambler plays against \(B\) with an initial fortune of \(N-k\) dollars.
  3. Each game he bets $1, wins with probability \(p\)
  4. Let \(\{X_t = t = 0,1,2 \dots\}\) represent his fortune as the betting goes on.
  5. Game only stops when either gambler or \(B\) is ruined.
  6. Here \(\mathcal{S} = \{0,1,\dots,N\}\)
  7. For a realization of the results of the first 10 games, (here \(p=1/2\)):
  sample(c(-1, 1), 10, replace=T)

Reference Textbooks

(Ross, n.d.; Pinsky and Karlin, n.d.)


Pinsky, Mark, and Samuel Karlin. n.d. An Introduction to Stochastic Modeling. Academic press.

Ross, Sheldon M. n.d. Introduction to Probability Models. Academic press.

Links to this note