In this article, we will cover:

• Major upgrade points of Gemma 3 over its predecessors

• Technical detail of Gemma 3

• Knowledge recap — knowledge distillation

• Knowledge recap — quantisation

• Benchmark performance of Gemma 3

• How to use image as inputs with Gemma 3?

• Gemma 1 and 2 recap

• Final thoughts

• Strong performance with practical model size

  • Gemma 3 is available in 1B, 4B, 12B, and 27B parameter.

  • Despite this small model size, it significantly outperforms many larger models (The 27B is currently ranked at the 12th place, outperforming Llama3-405B, DeepSeek-V3 and o3-mini!).

  • Gemma 3 can fit in one single GPU for on-device inference.

• The longest context window on the open-source market

  • The 4B, 12B, and 27B model support a 128K token context window, while 1B model supports 32K.

• Multi-lingual

  • Support 140+ languages!

  • This is a major improvement from Gemma 2, which only supports English text

• Multi-modal

  • Inputs to Gemma 3 can now include images! This enables Gemma 3 to be used for tasks such as visual Q&A, image captioning, and document analysis that includes images.

I also compiled a list of detailed breakdown for Gemma 3 for you. Enjoy!

Model Sizes

• 1B, 4B, 12B, 27B — all trained via knowledge distillation

• However, in the paper it is not specified what teacher models are used for knowledge distillation.

Context Window

• Extended to 32K tokens for 1B

• Up to 128K tokens for 4B, 12B and 27B

Multimodal + Multilingual

• Multimodal input via SigLIP vision encoder

• Multilingual support, a big shift from Gemma 1 & 2

Architectural Highlights

• Optimised local/global attention ratios

• Grouped-Query Attention

• Pan & Scan — a technique to reduce visual artifacts in image processing

Tokenisation

• Still based on SentencePiece, but now with:

  • Split digits (e.g., "123" → "1", "2", "3")

  • Preserved whitespace

  • Byte-level fallback encoding for rare chars

Quantisation-aware training

• Built-in QAT support for better low-precision performance

Further resources

Gemma 3 technical report

Gemma documentation

Sigmoid Loss for Language Image Pre-Training

SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

What is quantisation?

Common format: FP64, FP32, FP16, BF16, INT8

There are a few basic data types to know to understand quantisation in LLMs. In general, model parameters are encoded in a certain data format. The most common types are:

• Floating point — FP64 (64-bit), FP32 (32-bit)

• BFloat 16-bit (BF16)

• Integer 8-bit (INT8)

Floating point

The “floating point” format is one of the most common ways to represent numerical values in modern computers. The General structure:

Where:

• s = sign bit (0 for positive, 1 for negative)

• m = mantissa (fractional part)

• e = exponent

• bias is used to represent both positive and negative exponents

Here is a breakdown for each bit:

The total range supported by 64-bit, 32-bit and 16-bit are massively different:

• 64-bit: ~ ± 10^308

• 32-bit: ~ ± 10^38

• 16-bit: ~ ± 10^4

BF16

If we decrease the number of bits used in a floating point data format, the numerical range represented will vary. To keep the approximate same range for model training or inference, one can use formats such as bfloat16.

bfloat16 (BF16) is a 16-bit floating point format designed to provide speed and memory efficiency like FP16, while preserving much of the range and numerical stability of FP32.

It was introduced by Google and is widely used in deep learning hardware (like TPUs and newer CPUs/GPUs) because it strikes a great balance between performance and training stability.

Int8

Int8 (8-bit integer) is a compact numerical format that uses just 8 bits to represent whole numbers, typically ranging from -128 to 127 (signed) or 0 to 255 (unsigned). In practice, Int8 is used to represent floating point values approximately, using a transformation:

• Scale: the resolution (float step size)

• Zero-point: an offset that maps integer zero to the corresponding float

Post training quantisation

Post training quantisation (PTQ) is a technique used to convert a trained neural network (typically in 32-bit floating point precision, FP32) into a smaller and faster version by reducing the numerical precision of weights and activations — after training is completed.

• Step 1: model training

  • You train a model normally in FP32 for accuracy and stability.

• Step 2: quantisation

  • You convert:

    • Weights → to INT8 (or FP16)

    • Activations → either dynamically during inference, or using calibration data

• Step 3: calibration [optional]

  • You run a small sample of input data through the model to estimate ranges (min/max) for activations.

  • This improves accuracy by choosing better quantisation parameters.

Quantisation-aware training

Quantisation-aware training (QAT) is a technique where a neural network is trained to be aware of quantisation effects (often during training). QAT usually simulates quantisation during training to preserve accuracy.

⚙️ How it works

1. 🔁 Simulate quantisation during training

   • During forward passes, the model pretends weights and activations are low precision (e.g., INT8), but still stores and updates them in high precision (FP32).

   • This is done by inserting "fake quantisation" modules that simulate rounding and clamping to int ranges.

2. 🧰 Fine-Tune with quantisation noise

   • The model learns to adapt to the noise introduced by quantisation.

   • Back-propagation and weight updates still happen in full precision, but gradients reflect the quantised behaviour.

3. 🧠 Export fully quantised model

   • After training, you convert weights and activations to true lower precision (e.g., INT8).

   • The model is now fully quantised and optimised for deployment.

Further resources

A Visual Guide to Quantisation

IEEE754 wikipedia

What is knowledge distillation?

Knowledge distillation is a technique where a smaller, simpler model (called the student) is trained to mimic the output of a larger, more complex model (called the teacher).

In the LLM landscape, there are two common ways to perform knowledge distillation:

• Optimise the student model with token probability output from the teacher model as soft-label

• Optimise the student model with text dataset prompted with the teacher model

Further resources

• Distilling the Knowledge in a Neural Network G. Hinton et al. (2015)

• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

  Victor Sanh et al. (2019)

• A Survey on Knowledge Distillation of Large Language Models

  X. Xu et al. (2024)

  ---

Benchmark performance

For those who are not familiar with the common LLM benchmarks, I compiled a quick summary for you below. Enjoy!

Chatbot Arena

• One of the most referenced LLM leaderboards

• Human preferences on AI-generated outputs

• Evaluate and compare LLMs based on human preferences. Users can rank two AI-generated responses without knowing which models produced them

• Developed by researchers at UC Berkeley

Performance

Gemma 3 27B ranks 12th with an Elo score of 1339, outperforming larger open models like DeepSeek-V3, LLaMA 3.1 405B, and Qwen2.5-72B!

Further resources

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

Leaderboard

MMLU-Pro

• Language comprehension and reasoning

• Topics include Biology, Business, Chemistry, Computer, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology, Other

Performance

Gemma 3 27B achieves 0.676, outperforming Llama 3.3 70B!

This benchmark an enhanced version of the original Massive Multitask Language Understanding (MMLU) benchmark, designed to more rigorously evaluate the capabilities of large language models (LLMs) in language comprehension and reasoning across diverse domains.

An example of this benchmark is the following:

Question: Which of the following cases established the precedent that a defendant must be informed of the right to remain silent, the right to a lawyer, and protection from self-incrimination?​

Options:
A) Brown v. Board of Education​ 
B) Miranda v. Arizona​
C) Roe v. Wade
D) Betts v. Brady​
E) Plessy v. Ferguson​
F) Dred Scott v. Sandford​
G) Weeks v. United States​
H) Gideon v. Wainwright​ 
I) Marbury v. Madison​
J) Mapp v. Ohio​

Answer: B) Miranda v. Arizona​

Explanation: In the landmark case Miranda v. Arizona (1966), the U.S. Supreme Court ruled that individuals taken into police custody must be informed of their rights to remain silent and to have an attorney present during questioning. This decision established the "Miranda rights," ensuring protection against self-incrimination under the Fifth Amendment.

Further resources

MMLU-Pro leaderboard

LiveCodeBench

• This benchmark assesses code generation capabilities on real-world coding problems from platforms like LeetCode and Codeforces.

• There are four major areas: code generation, self-repair, test-output prediction and code execution

Performance

Gemma 3 27B achieves 29.7, while the score for Gemini-Flash-2.0-Exp are 31.8!

Further resources

Leaderboard

Bird-SQL

• Tests a model's ability to translate natural language questions into complex SQL queries across various domains.

Performance

Gemma 3 27B achieves 54.4, while the score for Gemini-1.5 are also 54.4!

Further resources

Leaderboard

GPQA Diamond

• This is a challenging dataset which comprises 448 multiple-choice questions across the domains of biology, physics, and chemistry, crafted by domain experts to ensure high quality and difficulty in PhD-level

Performance

Gemma 3 27B achieves 42.4, while the score for GPT-4o (0513) is 53.6%

Further resources

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Leaderboard

MATH

• Problem-solving, reasoning, mathematics

• A benchmark consisting of over 12,000 high school-level mathematical problems.

Performance

Gemma 3 27B achieves 89.0, while the score for Gemini 2.0 is 91.8

Further resources

Github

SimpleQA & FACTS Grounding

• Measures LLMs’ capacities to produce factual outputs as LLMs sometimes hallucinate

Performance

SimpleQA — Gemma 3 27B achieves only 10.0, while the score for Gemini 2.0 is 44.3!

FACTS Grounding — Gemma 3 27B achieves only 74.9, while the score for Gemini 2.0 is 82.8!

It can be seen that there is a significant performance gap between Gemma 3 and close-source models such as Gemini 2.0 for SimpleQA!

Further resources

SimpleQA

FACTS Grounding

How to use image as inputs with Gemma 3?

To use images as input with Gemma 3, Hugging Face provides a convenient way through the pipeline API using the "image-text-to-text" task. This allows you to pass a combination of images and text to multimodal variants of Gemma 3, such as gemma-3-4b-it, gemma-3-12b-it, or gemma-3-27b-it.

Here’s an example using a hosted image and a natural language prompt:

import torch
from transformers import pipeline

pipe = pipeline(
    "image-text-to-text",
    model="google/gemma-3-4b-it", # "google/gemma-3-12b-it", "google/gemma-3-27b-it" 
    device="cuda",
    torch_dtype=torch.bfloat16
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    }
]

output = pipe(text=messages, max_new_tokens=200)
print(output[0]["generated_text"][-1]["content"])

Gemma 1

If you would like remind yourself about Gemma 1, here are a quick recap:

• Model sizes: 2B and 7B parameters (both pre-trained & instruction-tuned)

• Context length: 8192 tokens

• Modality: Text-only, English-only

• Highlights:

  • Surpassed LLaMA 2 (7B & 13B) and Mistral 7B in many language tasks

Further resources

Gemma: Open Models Based on Gemini Research and Technology

Gemma 2

If you would like remind yourself about Gemma 2, here are a quick recap:

• Model sizes: 2B & 7B (via knowledge distillation), plus a 27B model (trained from scratch)

• Context length: Still 8192 tokens

• Modality: Text-only, English-only

• Architecture Highlights:

  • 🔹 Local + Global Attention Layers

  • 🔹 Grouped-Query Attention

• Tokenizer: SentencePiece

The 27B model in particular brought competitive performance with efficient scaling, while keeping the architecture relatively lightweight.

Further resources

Gemma 2: Improving Open Language Models at a Practical Size

Before you go, here are the takeaways:

• Gemma 3 is a major upgrade over Gemma 2

• A open-source model that enables full control for fine tuning, alignment, inference, or deployment.

• Competing performance, e.g. on benchmarks or human-based evaluations, even in comparison with models with much more parameters such as LLaMA 3, Mistral, and even close-source models (GPT or Gemini)

Resources

Gemma: Open Models Based on Gemini Research and Technology

Gemma 2: Improving Open Language Models at a Practical Size

Gemma 3 technical report

Huggingface blog on Gemma

This section of Data Science Letter highlights real-world use cases that reveal the power of applied machine learning (ML), artificial intelligence (AI), and data science (DS) across sectors.

Each post dives into a specific industry scenario — decoding the problem, the data, the algorithms used, and the business outcomes. Whether you're an aspiring data scientist or an industry professional, these articles are designed to bridge the gap between theory and practice.

Measuring Subscriber Value at Netflix: Causal Inference Meets Markov Chains link

Topic: machine learning, causal inference, Markov chain

Netflix doesn’t just acquire users — it quantifies the long-term value of each subscriber using advanced data science. This post explores how Netflix applies causal inference to isolate marketing impact and uses Markov chain modeling to predict retention behavior over time. Learn how these techniques guide smarter decisions in customer acquisition and lifecycle management.

Quick summary

Hello 👋 In this article, I will compare LightGBM, XGBoost and CatBoost in the following areas:

• Boosting algorithm

• Node splitting

• Missing data handling

• Feature handling

• Data sampling

• LightGBM-specific features

• XGBoost-specific features

• CatBoost-specific features

• Tips for choosing between LightGBM, XGBoost and CatBoost

• Resources

🚀 Email us @ social@verticalsolution.io if you want us to deep dive into a data science topic!

Boosting algorithm

Conventional boosting (LightGBM, XGBoost) vs Order boosting (CatBoost)

One of the major differences in tree building between LightGBM/XGBoost and CatBoost is the usage of ‘Order boosting’ in CatBoost.

In conventional boosting algorithms (used by LightGBM and XGBoost), at each boosting iteration, the tree is built using the same data points. It is argued that this repeated use of a single set of data points and can increase the chance of overfitting.

To mitigate this effect, CatBoost supports a different boosting algorithm as known as order boosting. The whole idea of this algorithm is to avoid repeatedly using same data points for both tree building and gradient or hessian computations. The method is briefly explained as follows:

1. First the original training dataset with size N is shuffled S times.

2. At each boosting iteration, for each shuffled dataset, a separate tree is built for each data position i (where i = 1, 2 ,…, N), using only data points before i (j < i).

3. The gradients and hessians for a particular data point k are then computed using trees built before k.

In reality, it is not practical to train a tree for each data position for each shuffled datasets, as the computational complexity would scale as SN². The actual algorithm builds trees for a fixed number of data positions K, reducing the complexity to SNK (usually K << N, one example could be K = logN).

In addition, by default CatBoost only performs order boosting when the input dataset size is small. For large datasets, CatBoost will use regular boosting algorithm instead.

Tree growing

Tree growing with LightGBM

LightGBM: leaf-wise / lossguide

In terms of tree growing methods, LightGBM uses leaf-wise tree growth (also called lossguide in other implementation). In this case, a tree is built leaf by leaf until a maximum number of leaves is attained. In each node splitting iteration, a non-terminal leaf with the best loss improvement is split.

Tree growing with XGBoost

XGBoost: depth-wise / lossguide

XGBoost supports both leaf-wise and depth-wise tree growth, called lossguide and depthwise respectively. In this case, a tree is built level by level until a specific depth is reached. Non-terminal leaves from the previous tree level are split according to each best loss improvement.

Tree growing with CatBoost

CatBoost: symmetric tree, depth-wise / lossguide

CatBoost supports three options: symmetric tree, depthwise and lossguide. The mode SymmetricTree is slightly different from depthwise. While the mode depthwise splits each node with its respective best loss improvement, SymmetricTree splits all node in the same depth with the same splitting criterion.

Node splitting

Node splitting with LightGBM and XGBoost

At each boosting iteration, boosting algorithm finds the optimal splitting position giving the best loss improvement for each node to grow trees.

In each node splitting, the difference in loss before and after splitting is computed:

The detail between each implementation relies on the definition of the loss function.

LightGBM and XGBoost uses primarily second-order approximations with regularisation for computing splitting gain.

In particular, for LightGBM, the splitting gain is defined as:

For XGBoost, the splitting gain is defined as:

Apart from regularisation, the formula of LightGBM and XGBoost look similar to one and other.

Node splitting with CatBoost

CatBoost supports four types of methods to compute splitting gains.

On the other hand, CatBoost supports four different splitting gain criteria, as known as following:

• L2, Cosine, NewtonL2 and NewtonCosine

One point to note is that the option of NewtonL2 is similar to the splitting gain definition in LightGBM and XGBoost. For interested users, please visit this page for more information.

Missing data handling

Missing data with LightGBM and XGBoost

During a node split, LightGBM and XGBoost assign the missing data to whichever direction giving the best split in loss.

LightGBM and XGBoost handle missing data by assigning these data points to whichever direction giving the best split in loss. This allows the model to learn from the patterns of missing data.

This feature is quite useful when data imputation or data filtering based on missing values are difficult to do, for example in situations where there are a lot of missing columns.

There is one thing to be mindful of. If there is no missing data in a column during training, but in inference there are missing data, then LightGBM and XGBoost will assign those data points to their respective default directions. Default directions can vary case-by-case. Visit these two discussions (here and here) on git if you are interested. In any case, in terms of model development, it is desirable to investigate on this train-test discrepancy as this can lead to significant degraded model performance from distribution drift.

Missing data with CatBoost

CatBoost either assigns all missing data to the left, or the right, or forbids the presence of missing data.

CatBoost does not provide an adaptive missing data handling as LightGBM or XGBoost. For numerical features, the following modes are supported:

• Forbidden — missing data is treated as an error

• Min — missing data for a feature is treated as the minimum value for that feature during a node split

• Max — missing data for a feature is treated as the maximum value for that feature during a node split

In general, one has to perform more manuel to handle missing data when using CatBoost.

Categorical Feature handling

Categorical features handling with LightGBM

LightGBM supports partitioning of categorical features.

LightGBM handles categorical features by partitioning its categories into 2 subsets and choose the subset which gives the best split, as shown in the figure below:

One advantage is that a tree will not need to grow very deep to achieve similar accuracy.

Categorical features handling with XGBoost

XGBoost supports partitioning of categorical features or one-hot encoding.

Starting from version 1.5, XGBoost supports both one-hot encoding and partitioning.

Categorical features handling with CatBoost

CatBoost transforms categorical features to numerical features by order target encoding.

One of the major differences between CatBoost and other boosting algorithm implementations is the way CatBoost handles categorical features.

Order target encoding — One major feature of CatBoost is to allow order target encoding. Normally, target encoding for categorical features can easily lead to data leakage as training labels are directly used as features. Instead, to reduce the risk of data leakage, CatBoost uses the same trick from order boosting datasets to compute target-encoded values for each data point.

For example, for binary classification problems, given a categorical feature with value as A, B, C, one example of the target-encoded value for A is:

where countInA is the sum of binary target with class A in the dataset, and totalCount is the total count of the dataset.

Data sampling

At each boosting iteration, it is possible to select only a fraction of the training datasets to grow the tree. There are a lot of ways to sample datasets. Examples include:

🔴 Uniform sampling means that each data point is independently sampled with a certain probability (a configurable parameter). For example, an uniform sampling of probability 0.3 will mean that at each iteration, roughly 30% of the training data will be used.

🔴 Parametric sampling refers to sampling methods with sampling weights defined by a parametric function, such as exponential or poisson functions.

🔴 Gradient-based sampling refers to sampling methods with sampling weights defined by gradients. The advantage is that data points with smaller gradient are in general well-trained, and hence should be downsampled. This can improve the training speed of the algorithm as some data points are dropped.

LightGBM, XGBoost and CatBoost supports different set of sampling methods. They will be explained below. Let’s start with LightGBM.

Data sampling method in LightGBM

LightGBM supports uniform sampling and a gradient-based sampling method called GOSS.

In LightGBM, the parameter to select sampling strategies is called data_sample_strategy. There are two options:

• bagging

• goss

Bagging refers to uniform sampling, with probability set by the parameter bagging_fraction.

Another option is called goss, which stands for gradient-based one-side sampling. In this method, top a% of data with the largest gradient values are kept, the remaining (1-a)% of data is downsampled. To avoid drift from original data distribution, the downsampled data are up-weighted in later process. This method ensures that we put more focus on the under-trained samples as the boosting iterations continue.

Data sampling method in XGBoost

XGBoost supports uniform sampling and a gradient-based sampling method.

In XGBoost, the parameter to select sampling strategies is called sampling_method. There are two options:

• uniform

• gradient_based

With the option uniform, one can set the probability with the parameter subsample.

The gradient-based sampling method supported by XGBoost samples data points according to the weight

where g and h are the gradient and hessian respectively, and lambda is the regularisation parameter. Note that this is a different gradient-based sampling method than the one supported by LightGBM above.

Data sampling method in CatBoost

CatBoost supports four types of sampling strategies: Bernoulli, Bayesian, Poisson and MVS.

In Catboost, the parameter to select sampling strategies is called bootstrap_type. There are four options:

• Bayesian

• Bernoulli

• MVS

• Poisson

Despite the difference in name, the Bernoulli sampling method is actually uniform sampling. One can control the probability of sampling with the parameter subsample.

The Bayesian sampling method samples data points with weights defined as:

where w is the weight for each data point. The parameter t is defined the bagging_temperature parameter. This is in fact the Bayesian Bootstrap method proposed by D. Rubin in 1981.

The poisson sampling method samples data points with weights defined as:

Finally, the MVS (Minimum Variance Sampling) method is a gradient-based sampling method. MVS is in some ways similar to GOSS in LightGBM. The differences between this method and the one in LightGBM are:

• A threshold is determined on the fly to determine above what values gradients are considered to be large, while in LightGBM a fixed percentage are pre-determined before training.

• Data samples with small gradients are weighted with a constant factor in LightGBM, while with MVS they are weighted with a factor proportional to the gradient value.

On theoretical basis, MVS usually gives a lower variance than GOSS. For interested readers, feel free to visit this paper for more detail.

LightGBM-specific features

Pairwise linear regression

Instead of using aggregated statistics, e.g. mean, median, in each leaf for predictions, LightGBM can perform linear regressions within each leaf and use these linear model to generate predictions instead. This function can be enabled by the argument linear_tree. This feature has been implemented since Dec 2020 (see the merge request here)

Exclusive feature bundling

This method aims to reduce the number of sparse features during tree growing by regrouping mutually exclusive sparse features into bundles. This feature aims to reduce memory usage and processing speed of the algorithm. Please refer to the original paper for more detail.

XGBoost-specific features

Survival-analysis

XGBoost supports custom loss functions for various survival analysis. One key difference between ordinary regression and survival analysis is that target labels in survival analysis can be are often represented by a range instead of point estimates, as the upper limit of the survival time can be infinite if the event has not occurred yet. Below shows an example of how to use define and use survival-analysis-specific loss functions in XGBoost.

import numpy as np
import os
import pandas as pd
from sklearn.model_selection import ShuffleSplit

import xgboost as xgb

# Read data
df = pd.read_csv('data.csv')
y_lower_bound = df['y_lower_bound']
y_upper_bound = df['y_upper_bound']
X = df.drop(['y_lower_bound', 'y_upper_bound'], axis=1)

# Train test split
rs = ShuffleSplit(n_splits=2, test_size=.7, random_state=0)
train_index, valid_index = next(rs.split(X))

# Construct train dataset
dtrain = xgb.DMatrix(X.values[train_index, :])
dtrain.set_float_info('y_lower_bound', y_lower_bound[train_index])
dtrain.set_float_info('y_upper_bound', y_upper_bound[train_index])

# Construct valid dataset
dvalid = xgb.DMatrix(X.values[valid_index, :])
dvalid.set_float_info('y_lower_bound', y_lower_bound[valid_index])
dvalid.set_float_info('y_upper_bound', y_upper_bound[valid_index])

# Perform training
params = {
  'verbosity': 0,
  'objective': 'survival:aft',
  'eval_metric': 'aft-nloglik',
  'aft_loss_distribution': 'normal',
  'aft_loss_distribution_scale': 1.20,
  'tree_method': 'hist',
  'learning_rate': 0.01,
  'max_depth': 6,
  'lambda': 0.01,
  'alpha': 0.02
}
bst = xgb.train(
  params, dtrain, 
  num_boost_round=10000,
  evals=[(dtrain, 'train'), (dvalid, 'valid')],
  early_stopping_rounds=50,
)

How to choose between LightGBM, XGBoost and CatBoost

Here are a few aspects to be considered when choosing between LightGBM, XGBoost, and CatBoost:

Memory storage and processing speed

LightGBM is in general a faster implementation among the three. If memory storage or training speed is a concern (e.g. when you have a dataframe using a large fraction of your memory already), it is probably good to try first LightGBM.

Missing data handling

LightGBM and XGBoost can assign data points with missing values that best optimise the objective function, while CatBoost does not support this feature. If your dataset has a lot of missing values and you would like to let the algorithm figure out its dependency by itself, it is probably best to use LightGBM and XGBoost.

Both LightGBM and XGBoost uses partitioning by default so it is not needed to do anything before training. (Note that this method only works for missing values in feature columns. In general it does not make sense to have missing values in target columns)

Overfitting

Regularisation parameters between the three implementations are more or less similar. In each implementation, there are a few common parameters that we can use to regularise the training and control overfitting, here are a few examples:

• Number of leaves ➡️ Maximum number of leaves in each tree (The higher this number, the more complex the trees will become)

• Maximum depth ➡️ Maximum number of depth in each tree (The higher this number, the more complex the trees will become)

• Data sampling fraction ➡️ The fraction of training data to be used in each boosting iteration. This reduces the likelihood of having a small portion of the training data dominating the training process and leads to overfitting

• Data sampling frequency ➡️ The frequency of which the training data will be sampled

• Feature fraction ➡️ The fraction of training data to be used in each boosting iteration

• Early stopping ➡️ Stop training whenever the performance does not improve for a while

• Lambda L1 or L2 ➡️ These few regularisation control the complexity of the trees (The higher these numbers, the less complex the trees are)

• dropout rate for DART (LightGBM or XGBoost) ➡️ During training, a fraction of trained trees are dropped at a certain probability. This helps regularise the whole algorithm as this removes over-dependency on a few trees.

For relatively small datasets with which overfitting is often a problem, one can consider to use order boosting in CatBoost.

Categorical feature handling

Both LightGBM and XGBoost support the functionalities to use one-hot encoding or partitioning for categorical features during node splitting, while CatBoost relies on order target encoding.

Target encoding works better when target labels have strong correlations with encoding categories. The disadvantage is that this technique is in general more prone to overfitting.

Linear model

Both LightGBM and XGBoost support linear models for predictions, instead of simple aggregated statistics, such as mean of target labels, from the leaves. This gives more flexibility for model developments.

Resources

Basics on gradient boosting

LightGBM

Documentation Paper

XGBoost

Documentation Paper

CatBoost

Documentation Papers

🚀 Thank you for reading! Give us some feedback @ social@verticalsolution.io

🚀 Subscribe to us for deep-dive content in DS, ML or AI! 🚀

Subscribe now

Table of content

1. Quick summary

2. Concepts for the algorithm

3. General and Catboost-specific parameters

4. Practical considerations

5. More resources

Quick summary

Catboost is another powerful library for gradient tree boosting, similar to LightGBM and XGBoost.

Compared to other boosting implementations (LightGBM and XGBoost), its special features are:

• Order boosting (a slightly modified gradient boosting algorithm)

• Advanced categorical variable handling

• Feature combination

• Symmetric trees

We will elaborate on these concepts in the following sessions.

Concepts for the algorithm

The following assumes that you are already familiar with how gradient boosting works. Otherwise for a brief review, please visit Concepts of boosting algorithms in machine learning.

Order boosting

One source of overfitting — Repeated use of the same dataset for each boosting iteration can lead to overfitting and hence prediction shift.

In many gradient boosting implementations (such as LightGBM and XGBoost), the whole training dataset is used to compute residuals, gradients or hessians based on the intermediate model from the previous boosting iterations. This repeated use of the same data points is one of the sources for overfitting.

Order boosting — a technique to build trees in each iteration by permutating training samples.

To mitigate this problem, Catboost uses order boosting.

First, the original training dataset is randomly permutated for S times. At each iteration, Catboost build a list of new trees for each data position i, using only data point before the position i.

This conceptual implementation scales with O(SN^2), where N is the number of data points. This is not practical for large datasets. In reality, Catboost applies order boosting when the number of training data is small. In addition, instead of maintaining a list of tree for all data position, it maintains a list from position i = 1, … , log(n) instead. This significantly reduces the complexity to O(SN) instead.

Advanced categorical variable handling

The implementation in Catboost to handle categorical variables are more elaborated than its counterparts.

Catboost supports several techniques to handle categorical variables, such as target encoding with ordering and feature combination.

Target encoding with ordering — One of the ways to handle categorical variables are by target encoding. However, this technique often leads to overfitting as the label values are used directly in feature preparation. Similar to the technique of order boosting in the previous section, Catboost provides a solution to this problem by computing target-encoded values (usually the average target values for a particular category) using the data points prior to the position considered, as shown in the figure below.

Feature combination — In addition, Catboost also combines different categorical features. For example, if there are two categorical features in the dataset: pet (with value as dog and cat) and color (with value as white, and black), Catboost would make new categories such as white dog, white cat, black dog, black cat. It is important to note that Catboost performs the categorical combination at each boosting iteration and dynamically adds categorical combination incrementally to avoid sparsity.

Feature combination allows more complex combinations of categorical values while maintaining reasonable training speed.

Symmetric trees

Unlike XGBoost and LightGBM, Catboost supports the feature to use symmetric trees as its base predictors. At each depth, Catboost ensures that all leaf nodes uses the same splitting rule. One advantage of this implementation is that this provides a faster inference speed. It also further avoids overfitting by constructing trees with simpler structures.

Symmetric trees — the splitting condition is the same at each depth

Advanced bootstrapping strategy

Compared to LightGBM and XGBoost, Catboost supports a wider range of bootstrapping strategies during tree building, which includes

• Bayesian, Bernoulli, MVS, Poisson

One can also define the sampling unit and sampling frequency.

On the other hand, LightGBM and XGBoost support

• Bagging, GOSS (For a description on GOSS, please visit here)

• Uniform, Gradient-based

respectively.

Parameters

General parameters (similar to LightGBM and XGBoost)

Catboost supports various parameters to control and tune training processes specific to boosting.

These parameters have similar counterparts in other implementations (LightGBM, XGBoost), and their meanings are easily understood given a basic understanding of gradient boosting (Please visit here for a deep-dive into gradient boosting).

The most common ones are as follows:

• loss_function — the training objective for the problem

• learning_rate — the numerical factor multiplied to each predictor for each boosting iteration

• iterations — total number of boosting round to be performed.

• random_seed — initial random seed used for training.

• l2_leaf_reg — L2 regularization term during tree building

• subsample — bagging rate during tree building, used when the bootstrap type is Poisson, Bernoulli and MVS.

• colsample_bylevel — fraction of features to be used during each tree split.

• min_data_in_leaf — minimum number of data required in each leaf.

• max_leaves — maximum number of leaves

• grow_policy — a parameter to choose between

  • SymmetricTree (All leaves from previous depth are splitted with the same condition)

  • Depthwise (All non-terminal leaves are splitted with different conditions)

  • Lossguide (The non-terminal leaf with the best loss improvement is split)

• leaf_estimation_method — The method used to calculate the prediction from a leaf. It is a parameter to choose between:

  • Newton

  • Gradient

  • Exact

Specific parameters to Catboost

bootstrap_type

Catboost supports a wide variety of sampling methods during splitting. These methods control the weights used for each training sample at each split, This can partly reduce the effect of overfitting during training.

Specifically, Catboost supports the following parameters:

• Bayesian — the weight of each training sample is calculated by:

  where φ is a random number sampled from the uniform distribution [0,1]. The parameter t is the bagging temperature and can be defined by setting bagging_temperature in Catboost.

• Bernoulli — this is the typical bagging method also used in LightGBM and XGBoost. The rate of sampling is defined by the parameter subsample as described in the previous section

• MVS — Similar to GOSS in LightGBM, this is a bagging method which samples training samples based on gradient values. For more detail about this complex algorithm, please visit the paper here.

• Poisson — the weight of each training sample is calculated by:

• No

Please note that one can also control the frequency of sampling with the parameter sampling_frequency:

• PerTree — weights are determined at each construction of new tree

• PerTreeLevel — weights are determined at each split

nan_mode

This is a parameter to choose the strategy to handle missing data during training. While there are default ways to handle missing data in LightGBM, in Catboost we can choose explicitly this strategy.

In particular the following options are provided:

• Forbidden — does not allow missing value, and raise error when encounter so.

• Min — put all missing values during a split to the left side

• Max — put all missing values during a split to the right side

boosting_type

This is a parameter to choose the boosting algorithm.

In particular, one can choose between:

• Ordered — Choosing this will enable Order Boosting (as explained in the previous sessions)

• Plain — This will enable the classic gradient boosting scheme.

  ---

Practical considerations

Catboost is in general slower but performs better with small datasets, and categorical variable handling is more elaborated.

Compared to LightGBM and XGBoost, the speed of Catboost is generally slower. With small datasets, where the processing speed is not the primary concern, using Catboost with Order Boosting enabled could give a better performance.

In addition, Catboost can give significant advantages over its counterparts in situations which the effects of feature interactions are significant. It can also be a good choice with training data with complex categorical variables.

Conclusion

In many ways, Catboost is similar to LightGBM and XGBoost. In many ways, it is different because there are additional features built in this package that its counterparts do not have. This article highlights these differences and hope to help you feel more comfortable when using this package. Stay tuned for our next deep-dive article!

More resources

Paper from Catboost [1] [2]

Videos on Catboost

🚀 Subscribe to us for deep-dive content in DS, ML or AI! 🚀

Subscribe now

🚀 Subscribe to us @ newsletter.verticalsolution.io for more content in DS, ML or AI! 🚀

Read more

🚀 Subscribe to us @ newsletter.verticalsolution.io for more content in DS, ML or AI! 🚀

Subscribe now

Causal inference

This session covers various topics on causal inference and observational studies in data science.

Experimentation

This session covers various topics on experimentation and AB testing in data science.

🚀 Subscribe to us @ newsletter.verticalsolution.io for more content in DS, ML or AI! 🚀

Subscribe now

• Randomised control trials (RCT) [covered in this article]

• Covariate adjustment in RCT

• Regression analysis

• Propensity score matching

• Propensity score weighting

• Double-robust estimators

• Subgroup analysis

• Double machine learning

• Instrumental variable

• Difference-in-difference

• Regressio…

Read more

🚀 Subscribe to us to more deep-dive topics on data science, machine learning and artificial intelligence!

Subscribe now

This article highlights boosting algorithms in scikit-learn. For a conceptual introduction to gradient boosting, please visit our article here.

Introduction

In essence, scikit-learn offers two options for gradient-boosted trees:

1. Gradient boosting

2. Histogram-based gradient boosting

For gradient boosting, one can access the class:

# Gradient boosting
GradientBoostingClassifier
GradientBoostingRegressor

while for histogram-based gradient boosting, one can use:

# Histogram-based gradient boosting
HistGradientBoostingClassifier
HistGradientBoostingRegressor

Each method will be explained in detail below in separate sessions.

It is useful to note the pros and cons when using each of the options.

In terms of speed, histogram-based gradient boosting is generally faster than gradient boosting as the node splitting complexity of gradient boosting is proportional to the number of data points and the number of features.

In terms of accuracy, as the node splitting procedure is coarser in histogram-based gradient boosting, gradient boosting generally could develop a more complex model as the node splitting is finer.

The histogram-based implementation in scikit-learn supports handling of missing values and categorical data, making upstream data preprocessing steps a bit easier.

Gradient boosting

The class GradientBoostingClassifier and GradientBoostingRegressor implemented in scikit-learn use the classic gradient boosting algorithm described in our article here.

In particular, at each boosting round, a regression tree DecisionTreeRegressor is fitted using the negative gradients of the loss function as target labels (the gradients are computed at prediction values from the previous iterations). The objective of the regression tree is either mean square error (MSE) or MSE with improvement score by Friedman. This objective is also used to compute the quality of each split.

It is important to note that other implementations such as LightGBM or XGBoost have different tree fitting procedures.

These two classes are relatively slow when the training dataset is large, as each node split is determined by looking at all possible split for all features.

Histogram-based gradient boosting

Inspired by LightGBM, scikit-learn also implemented histogram-based gradient boosting.

During tree fitting, the training samples are first binned into integer-valued histograms (the default value for the number of bins is 256). Then for each node splitting these histograms are used instead of considering all possible splits for each features. This usually speeds up training time as there is no need to sort dataset for each feature for each boosting round.

Also like LightGBM, this class also support handling of missing values. In particular, during training, samples with missing values would be assigned to the node (left node or right node) which maximises the potential gain from the node splitting. During inference, the samples with missing values will be assigned to the same node as in the training accordingly. If no missing values are encountered during training, the samples with missing values will be assigned to the node with more samples.

Summary

In conclusion, this article explored the concept of boosting algorithms within the scikit-learn library.

We reviewed the foundations of boosting, a technique for creating powerful ensembles by sequentially training weak learners. We then delved deeper into gradient boosting, a popular boosting method that builds an additive model by fitting decision trees on the negative gradients of a loss function.

Finally, we introduced histogram-based gradient boosting, a faster variant particularly suitable for larger datasets.

By understanding these boosting techniques in scikit-learn, you can leverage their ensemble power to enhance the accuracy and robustness of your machine learning models!

Consider subscribing to our paid version with a price of one cup of cappuccino per month if you find our content useful!

Subscribe now

What is bootstrapping?

The idea of bootstrapping is that statistical inference about a population from a sample data can be approximated by two steps:

1. resampling the sample data

2. statistical inference from resampled data

In this following sections we will talk about various types of resampling and statistical inference methods in bootstrapping.

Practical considerations when using bootstrapping

Statistical inference involves drawing statistical conclusions from a finite data sample and extrapolate these conclusions to the whole population.

However in reality, even drawing statistical conclusions from a finite data sample could be tricky in the following ways:

1. We do not know a-priori the correct probability density function (pdf) for our chosen test statistics.

2. Even if we know the correct pdf for our test statistic, it may not be in a nice analytical form.

Bootstrapping is particularly advantageous in scenarios where either an analytical expression for the sampling distribution is unavailable or the application of asymptotic theory (e.g., central limit theorem) is uncertain.

If we have a large sample data, this sample can in fact approximate the true underlying pdf.

It is also worth noting that the quality of inference from the resampled data could be known as we can compare these inference results with the ‘true’ population (the original sample data). This could provide a proxy for the inference result that we would like to know for the true population.

Resampling methods — Case bootstrapping 

Case bootstrapping is probably the simplest resampling methods. This method involves drawing samples with replacement from the original data set:

Assuming the original data set has n observations

1. k resampled data with n observation (also called bootstrap sample) will be formed by sampling with replacement, where k is often between 50 to 1000.

2. The test statistic of interest is calculated for each resampled dataset.

In the end, A distribution of the test statistic of interested (often called the Bootstrap distribution) with size k is generated. With one of the statistical inference method explained below, this distribution can be used to perform hypothesis testing or confidence interval estimation.

Resampling methods — Bayesian bootstrapping  

This method is first proposed in this paper in 1981.

Instead of actually sampling the original data set, Bayesian bootstrapping creates bootstrap samples through reweighing the original data set:

Assuming the original data set has n observations

1. Generate a list of random number, let’s call this list w, in the interval [0,1] with size n-1, sort this list and append 0 and 1 in the start and the end.

2. Reweigh each data point by the following formula:

   where w(0) = 0 and w(1) = 1

3. The test statistic of interest is calculated for this reweighed data set.

4. Repeat this process k times, where k is often between 50 to 1000 (similar to what we have in case bootstrapping)

Resampling methods — Poisson bootstrapping 

This method is explained in, for example this paper from Google in 2012.

The idea of this method is based on the intuition that the procedure of case bootstrapping is actually equivalent to assigning a weight vector to each data point, where the weight vector is drawn from multi-binomial distribution:

It is worth noting that in the limit when N goes to infinity, we have:

This means that for large dataset, bootstrapping with weights iid-drawn from Poisson(1) will give a similar result as with the case bootstrapping (sampling with replacement).

The advantage with Poisson bootstrapping is that it is not necessary to know the total number of data points in advance during bootstrapping. Hence this method can be easily parallelised in distributed computing environment such as spark.

Statistical inference — Percentile method

The basic method is very simple. After getting the bootstrap distribution of the test statistic, one can simply compute the empirical quantiles and use these as the confidence interval:

where α is the confidence level. In addition, there is also another method which uses the reverse percentile as the confidence interval estimation, i.e.

It is worth noting that this method is recommended when the bootstrap distribution does not have a long tail and is mostly symmetric.

Statistical inference — Studentised method

In general, the studentised method is a more accurate method, but comes with a larger computational cost, as it involves two-step bootstrapping This is how it works:

1. Construct a set of bootstrap sample:

2. Calculate the test-statistic for each bootstrap sample as other methods

3. For each bootstrap sample, i.e. b = 1, 2, 3, … , B,

   1. Construct another set of bootstrap sample:

   2. Calculate the test-statistic for each new bootstrap sample

   3. Calculate the standard deviation based on these M test-statistics

4. Calculate the t-statistic for each bootstrap sample:

5. Construct α/2 and 1-α/2 quantile from the boostrapped t-statistic distribution

6. The confidence interval can be calculated as:

Key takeaways

Advantages of bootstrapping:

• It is quite general and can be applicable to a wide range of statistical inference problems

• It is conceptually simple to be used to estimate statistical quantities, such as standard errors and confidence intervals, even for very complex test statistics

• It can also incorporate different sampling methods easily

Disadvantages of bootstrapping:

• It can be computationally intensive

• It is subject to the quality of the original data sample

• As an empirical method, its assumptions and validity are harder to be examined

Summary

This article summarises a few common methods in bootstrapping, a statistical method for estimating sampling distributions. We explored various resampling methods, such as case bootstrapping, Bayesian bootstrapping and Poisson bootstrapping. We have also talked about a few methods to perform confidence interval estimation given a set of bootstrap samples.

If you find that our articles are useful for your machine learning journey, consider subscribing to our paid version with a price of one cup of cappuccino per month.

Subscribe now

Why multi-armed bandit?

Compared to typical machine learning, (non-contextual) multi-armed bandits, also as known as MAB, have several practical advantages:

1. Light model serving — Due to model simplicity, MAB typically does not require a separate model call to obtain predictions. In addition, MAB does not require features. It is much easier to serve a MAB model as resources are not needed to build up a feature engineering pipeline and suitable model architectures.

2. Easy maintenance — Due to its little dependency on other parts of the pipeline (such as feature store, model prediction call), a MAB solution is typically much easier to maintain.

3. Handle of data drift — Customer behaviours change over time. MAB is typically used as an online learning algorithm, making it a natural solution to handle data drift.

Stochastic Bandit

Let’s start with one of the simplest form of bandits.

Problem statement

Given a fixed number of round T and K arms:

For each round:

1. A policy (or a model, an algorithm) will select one arm, hence performing an action.

2. Based on this action, the reward for this action is sampled from the corresponding reward distribution.

3. The model uses this reward to update itself.

The process repeats T times.

Assumptions

In addition to the formulation, there are two major assumptions in this simplest form of MAB:

1. We can only observe rewards for the chosen arm at each round

2. The reward distribution of each arm is independent of each other. When an arm is chosen, the observed reward is sampled from the reward distribution of the chosen arm. Rewards in each round are hence iid.

Care has to be taken for Assumption 2. For data science problems in which data drift is an issue, it may be necessary to:

• use other MAB algorithms which could approximately account for drift in data or reward distribution. For example contextual bandits with time-dependent or seasonal features. For an introduction in contextual bandit, please visit our articles here.

• Maintain a certain amount of exploration, such as epsilon-greedy algorithm. Concepts of exploitation and exploration will be described in the next section.

Exploitation and exploration

In multi-armed bandits (or in reinforcement learning), exploration and exploitation represent two completely opposite strategies.

Given a bandit problem, exploitation represents choosing the best arm based on current knowledge of the system (which may be incomplete or misleading), while exploration means trying out suboptimal arms to collect extra information about the system.

It is important to note that finding the balance between exploitation and exploration is one of the most crucial points in multi-armed bandit problems. Various algorithms have been designed to tackle this point. In the following sections we will introduce a few of them.

Epsilon-greedy algorithm

Definition

Given a MAB problem with total number of round as T and K arms, the epsilon-greedy algorithm is as follows:

For a given value of epsilon, the epsilon-greedy algorithm is as follows:

For each round t = 1, 2, 3, …, T,

1. draw a random real-valued number r in the interval of [0,1]

2. To choose an arm, we do:

   1. If r is less than epsilon, choose an arm randomly

   2. If r is greater than epsilon, choose the arm with the highest average reward

3. Observe the reward with the chosen action, update the average reward for the chosen arm accordingly.

Key takeaway

1. Data draft: For a constant epsilon, this algorithm implies that there is always a certain degree of exploration. In case of data drifts, data collected from this constant exploration could be helpful to detect data drift and mitigate some of its bias. However, in a situation with no data drift, this constant exploration could be suboptimal.

2. Non-adaptive exploration: The algorithm uses non-adaptive exploration. Alternatively, we can adjust epsilon according to the number of played rounds, the number of arms or other parameters. So that the exploration schedule is adapted to the history of observed rewards. There are other MAB algorithms with adaptive exploration schedules, such as UCB (which is explained below).

UCB algorithm

Upper confidence bound (UCB)

Before introducing the UCB algorithm, it is necessary to define UCB (upper confidence bound):

UCB for a given arm a at a given round t is defined as:

where μ is the average of all observed rewards for arm a up to t round, n is the number of times that arm a has been selected up to t round.

Note that the latter term in the expression is an estimate of the confidence interval of the expected reward for arm a up to t round. Derivation of this confidence interval is based on the Hoeffding Inequality, which is explained in Appendix.

We are in the position to define the UCB algorithm.

Definition

Given a MAB problem with total number of round as T and K arms, the UCB algorithm is as follows:

For each round t = 1, 2, 3, …, T,

1. Pick the arm with the highest upper confidence bound (UCB)

The intuition behind the UCB algorithm is that, there are two scenarios which we would like to select an arm and observe its rewards:

1. The average reward of this arm is higher than others arm. (Exploitation)

2. We have not selected this arm enough times to be confident about its average reward. (Exploration)

The first case will give a high value in the first summand in the UCB formula, while the second case will mean a high value in the second summand. Therefore, picking the arm with the highest UCB represents one natural way to balance exploitation and exploration.

Key takeaway

• The confidence bound in the UCB algorithm is derived without any prior information on reward distributions. Hence the confidence bound would be a bit conservative.

• In case when there are more knowledge a priori on reward distributions (for example they follow probability density functions, such as normal distribution), we can replace the UCB bound with specific confidence bounds derived from this pdf instead.

• The UCB algorithm represents one of the most simplest algorithms which takes into account the uncertainty on average reward estimate for each arm. In this algorithm we use UCB as a heuristic to balance exploitation and exploration. There are other UCB variants which further expands on this idea.

Bayesian UCB algorithm

Bayesian UCB is first proposed in this paper.

Unlike UCB which uses a specific formula based on the Hoeffding bound, to estimate confidence bounds, Bayesian UCB provides a more general mathematical framework to include custom priors and perform Bayesian updates on model parameters per round.

For a more general introduction on Bayesian statistics, please visit this article in our data science section.

Definition

For each round t = 1, 2, 3, …, T,

1. Compute the 1-1/t quantile for each arm as below, where Q is the quantile function with the first argument as the confidence level, and λ as the posterior distribution for mean reward for arm j for round t-1.

2. Choose the arm corresponding to the highest quantile value

3. Observe reward and perform Bayesian update (posterior estimation) on λ for the chosen arm.

Key takeaway

The main advantage of this algorithm is that it is a simple policy as UCB, but still generic enough to incorporate various Bayesian statistical models. This algorithm also has a proper Bayesian interpretation.

In addition, one can choose probability distribution from the one-parameter exponential family (such as normal distribution, Bernoulli distribution or gamma distribution) as their conjugate to simplify the procedure to estimate posterior distributions.

Thompson sampling

Thompson Sampling is one of the most commonly-used bandit algorithms. It is proposed in this paper by Thompson in 1933.

Like Bayesian UCB, it is also a Bayesian bandit algorithm. Let’s look at what this algorithm does.

Definition

For each round t = 1, 2, 3, …, T,

1. For each arm, sample its posterior distribution and get a sample of mean reward

2. Choose the arm with the highest sampled mean reward

3. Observe the resulting reward from this chosen arm

4. Perform Bayesian updates with this extra pair of data (chosen arm, received reward)

Key takeaways

Although both Thompson sampling and Bayesian UCB are Bayesian bandits, there are some similarities and differences between the two algorithms.

Similarities

1. At each round, after receiving a new pair of data point (chosen arm, received reward), the posterior distribution is updated and used as the prior for the next round.

2. Both algorithms can explore because the posterior distributions are not deterministic. The less uncertain the posterior distribution is, the more exploration that would be induced in both algorithms.

3. Both algorithms require a specification on initial prior distributions. A careful choice of prior can improve performance and convergence rate.

Difference

1. Bayesian UCB tends to be more optimistic in the beginning due to the schedule of the quantile 1-1/t.

2. Bayesian UCB uses an aggregated test-statistic, i.e. the (1-1/t)-th quantile of the posterior distribution, while the Thompson sampling uses a sampled value from the posterior distribution only.

Summary

In this article we have covered basic concepts of stochastic multi-armed bandits and a few commonly used MAB algorithms, including epsilon-greedy algorithm, UCB, Bayesian UCB and Thompson sampling. We have talked about the basic concepts of each algorithms and some practical considerations when using them.

However, there are a lot more to talk. Please stay tuned to our in-depth guides for each algorithm (and more)!

If you find that our articles are useful for your data science journey, consider subscribing to our paid version with a price of one cup of cappuccino per month.

Subscribe now

Before going into the detail of sampling and selection bias, we need to clarify the concept of population and sample in statistics. Those who are familiar with the concept can skip the next section

Population and sample

To put it in very simple words, a population is the whole set of statistical unit (such as all people living in US when conducting an US income survey) that we would like to draw statistical conclusion on.

However, in reality, it is impossible to survey or gather data for every statistical unit. Therefore, we often sample a finite number of statistical unit and perform statistical inference on this finite group to infer information on the whole population. The finite group is a sample.

It is important to note that since we are effectively extrapolating conclusions drawn with a sample to a population, it is important to respect a few statistical concepts, sampling and selection bias are one of these concepts to respect. This ability of extrapolation is called external validity in statistics.

Random sampling

Random sampling constitutes a selection process where every element in the target population possesses an identical probability of inclusion in the sample at each draw. The resulting collection is termed a simple random sample.

This process can be implemented with replacement, where observations return to the population pool after each selection, allowing for potential re-selection in subsequent draws. Alternatively, random sampling can be conducted without replacement, where chosen observations are no longer available for future selections.

Random sampling is the simplest form of sampling. It requires a complete sampling frame (i.e. access to all statistical units), which could be or not be available. This sampling method is also a representative of the underlying population.

Selection bias

Selection bias occurs when statistical units are selected (or sampled) from a population in such a way that randomisation is not properly realised, hence the obtained sample is not a representative of the population.

While there are many types of selection bias in statistics, in this article we will explain a few of them below.

Sampling bias

Sampling bias occurs when certain statistical units are selected consistently than other units. Note that sampling bias is one of the most common ways leading to selection bias.

Early stopping

Selection bias can arise from early termination of an AB test at a time when its results support the desired conclusion.

Data handling

Selection bias can also arise from filtering on a sample obtained from a randomised trial based on certain data properties, such as rejecting bad data

Cherry-picking

Selection bias can also occurs when a subset of sample is specifically selected to support a ‘conclusion’, or when a set of studies are specifically selected when conducting meta-analysis.

Summary

While in some cases it is trivial to avoid selection bias, in other cases it is impossible to avoid them. It is important to understand that such bias exists and how it could even affect conclusions drawn from properly-designed experiments.

About this post

This post explains practical knowledges when using LightGBM.

For a more conceptual introduction of boosting, please visit another post ‘Concepts of boosting algorithms in machine learning’.

For an introduction on XGBBoost, please visit another post ‘Practical guides on XGBoost.

Introduction

Like XGBoost, LightGBM is a powerful implementation of tree-based boosting algorithm.

This article explores the core features of LightGBM, focusing on gradient-based sampling and exclusive feature bundling, and practical considerations with this open source package. By reading this article, you can understand the essence of LightGBM and its differences to other implementations.

The official website for LightGBM can be found here. Its corresponding paper can be found here.

Core features

Newton tree boosting

LightGBM uses Newton tree boosting!

Like XGBoost, LightGBM uses Newton descent at each boosting iteration, although the corresponding paper for LightGBM does not mention the usage of Newton descent. From the implementation on github (search for the static double function GetLeafGain), it can be seen that the leaf gain in a node splitting is evaluated as:

where apart from regularisation parameters, are the same as the formula implemented in XGBoost.

Please refer to the article ‘Concepts of boosting algorithms in machine learning’ for an introduction to gradient tree boosting and Newton boosting.

Gradient-based one side sampling (GOSS)

GOSS is a way to speed up training time by sampling data based on gradient values!

Specifically, GOSS firstly sorts the data points with the absolute value of their gradients. The top a × 100% points are selected as part of the training data. In addition, GOSS also randomly samples b × 100% instances from the remaining of the data (these are data points with smaller gradients). To compensate for the difference in sampling, GOSS reweighs the sampled data with small gradients by a constant (1−a) / b when calculating the information gain.

This method helps reduce computation time by reducing the number of data points used during tree growing without changing the original data distribution by much.

Node splitting for categorical features

Node splitting with categorical variables are often better handled in LightGBM than with typical methods like one-hot encoding!

One common approach to encode categorical features is one-hot encoding. It is often observed that this approach is not very optimal for tree learning algorithms, especially with categorical features with high cardinality.

LightGBM instead handles categorical features natively. Specifically at node splitting, a categorical feature is split by dividing its categorical into two subset. There are 2^k possible ways to do so. The quality of split is then evaluated for each partition and the best split is then chosen.

This is in many cases a better approach than one-hot encoding. At each split all categorical values are considered and handled, while with one-hot encoding only one categorical value will be handled at each split.

Exclusive feature bundling

This method aims to reduce the number of sparse features during tree growing by regrouping mutually exclusive sparse features into bundles. The bundle of exclusive features into a single feature is called an exclusive feature bundle.

Interesting features

Pairwise linear regression at each leaf

Instead of using the average values in each leaf for predictions, LightGBM can perform linear regressions within each leaf and use these linear model to generate predictions instead. This function can be enabled by the argument linear_tree.

Since Dec 2020 (merge request here), it is possible to use piecewise linear regression in LightGBM. This allows practitioners to use linear regressions at each leave for predictions (instead of using aggregated statistics of target labels).

The implementation is slightly different from the paper ‘Gradient boosting with piece-wise linear regression trees’ (link). In particular in LightGBM, during node splitting, the splitting is determined in the same way as without linear models. After the tree structure has been determined, a linear regression model is fitted in each leaf. In the original paper the splitting also include fitting a linear regression model, leading to a much more computationally intensive procedure.

Tackling overfitting with LightGBM parameters

A common problem from gradient boosting is overfitting. With LightGBM, one can reduce the variance of the model with the following parameters:

• Decrease max_bin

• Decrease num_leaves

• Decrease max_depth

• Increase lambda_l1, lambda_l2

• Increase min_data_in_leaf

• Increase min_sum_hessian_in_leaf

• Enable data or feature bagging with bagging_fraction, bagging_freq or feature_fraction

Hyperparameters

While there are many hyperparameters to be tuned in LightGBM, this session explains the most important ones that we would encounter a lot in data science or machine learning problems.

boosting

Similar to the argument booster in XGBoost, this argument allows you to choose what boosting algorithm to use, rf, gbdt and dart.

learning_rate

This argument corresponds to learning rate or shrinkage rate. Please refer to our post Concepts of boosting algorithms in machine learning if needed.

num_iterations

This argument controls the total number of boosting iteration to be carried out.

max_depth

This argument refers to the maximum depth that a weak tree can have at each boosting iteration. The higher this value is, the more complex each weak tree could become, and more likely it is for the final model to overfit.

bagging_fraction

This argument refers to the fraction of rows randomly-sampled for weak tree training at each boosting iteration. This sampling can help reduce overfitting.

linear_tree

This argument controls whether to run piecewise linear regression at each leaf or not. Please refer to previous session for an introduction of the method.

lambda_l1, lambda_l2

These two arguments correspond to the two regularisation hyperparameters to control the size of the weak tree.

drop_rate

This argument is only relevant when boosting is dart. This refers to the probability of dropping a weak tree during a boosting iteration.

Summary

In this article we talked about the algorithm of node splitting in LightGBM, two methods that enables its speed-up during boosting and an interesting feature called pairwise linear regression and a list of important parameters to pay attention to during training. We will continue writing more deep-dive content on this topic so stay tuned!

About this post

This post explains practical knowledges when using XGBoost. For a more conceptual introduction of what XGBoost is under the hood, please visit another post ‘Concepts of boosting algorithms in machine learning’.

Introduction

Extreme Gradient Boosting, XGBoost, is a powerful implementation of tree-based boosting algorithm in python. This note explores its core features of XGBoost, focusing on regularised learning objectives, gradient tree boosting, learning rate parameter, row and column subsampling techniques, and node splitting algorithms. By reading this note, you can understand the essence of XGBoost and its differences to other implementations.

The official website for XGBoost can be found here. Its corresponding paper can be found here.

Core features

While there are a number of key improvements in XGBoost, in this post we focus on the two most important ones: gradient tree boosting with regularisation and approximate node splitting algorithm.

Gradient tree boosting with regularisation

Like gradient tree boosting, at each boosting iteration, a weak tree is fitted and then added to the final predictor.

Specifically, when growing a weak tree, the optimal split at each node is determined when there is a maximum loss reduction as follows:

where

where k represents left, right or no-split,

represents the gradient and hessian for data point j respectively,

represents the regularisation hyperparameters to control the size of the weak tree.

This formula is derived by a minimisation of a second-order Taylor approximation to the regularised loss function. You can find the exact derivation in the original paper.

Node splitting

To find the best split for a given node during tree fitting, a split finding algorithm is needed to determine how to split the data points in this given node to left and right.

In particular, XGBoost provides supports to the exact, approximate and histogram-based splitting algorithm.

Exact algorithm

This algorithm enumerates over all features, and for each feature it iterates over all values to find the split that would maximise the loss reduction (defined in previous section). This algorithm is very computationally intensive. To slightly optimise for performance, XGBoost first sorts the data points with respect to feature values and accumulate gradient and hessian statistics accordingly.

Approximate algorithm

Instead of considering all possible splits from all features, the approximate algorithm iterates on a subset of splits for each feature only.

It is possible to propose candidate splits before and during tree fitting. The former, as known as global splitting, constructs candidate splits before any node splitting procedures. The latter, as known as local splitting, constructs candidate splits for each node splitting. XGBoost supports both methods.

Histogram-based algorithm:

This candidate splitting method is similar to the one used by LightGBM. Refer to practical guide on LightGBM in the same series to know more about this algorithm.

Missing data

In presence of missing data in training data, XGBoost handles this by assigning missing data to the side (left or right) such that the loss reduction is maximal.

Hyperparameters

While there are many hyperparameters to be tuned in XGBoost, this session explains the most important ones that we would encounter a lot in data science or machine learning problems.

booster

This argument allows you to choose what boosting algorithm to use, gbtree, dart and gblinear. Both gbtree and dart use tree as weak learners, while gblinear uses linear models. The difference between gbtree and dart is that dart includes a random dropout during training phases, which randomly masks weak learners with a certain probability, to further avoid overfitting.

eta

This argument corresponds to learning rate or shrinkage rate. Please refer to our post Concepts of boosting algorithms in machine learning if needed.

gamma, lambda

These two arguments correspond to the two regularisation hyperparameters to control the size of the weak tree, as explained in the previous session of this post.

max_depth

This argument refers to the maximum depth that a weak tree can have at each boosting iteration. The higher this value is, the more complex each weak tree could become, and more likely it is for the final model to overfit.

max_leaves

This argument refers to the maximum number of leaves that a weak tree can have in each boosting iteration.

subsample

This argument refers to the fraction of rows randomly-sampled for weak tree training at each boosting iteration. This sampling can help reduce overfitting.

colsample_bytree, colsample_bylevel or colsample_bynode

These three arguments controls sampling of columns for weak tree training. This sampling can happen for each tree (bytree), for each tree depth (bylevel) or for each node (bynode).

sampling_method

This argument specifies the sample weight used for training data points. XGBoost supports:

• uniform sampling — each data point has an equal weight

• gradient-based sampling — each data point is sampled proportional to the weight

tree_method

This argument specifies the node splitting algorithm to be used. Node splitting algorithms supported by XGBoost are explained in the previous section Node splitting.

🚀 Subscribe to us for more deep dive content on data science, machine learning and artificial intelligence!

Subscribe now

Statistics

This section talks about essential statistics concepts in data science, focusing on how these concepts relate to real-life scenarios.

Causal inference

This section talks about essential concepts in causal inference, focusing on how these concepts relate to real-life scenarios.

Machine learning

For more ML-related content, please visit our article list on machine learning here.

If you find that our articles are useful for your machine learning journey, consider subscribing to our paid version with a price of one cup of cappuccino per month.

Subscribe now

⚡ Large Language Models (LLMs)

Large Language Models are transforming the landscape of AI, powering everything from chatbots and code assistants to document summarization and creative generation. In this section, we explore the theory, engineering, and practical application of LLMs—along with guides to prominent open-source and proprietary models.

• 🛠️ Gemma 3 Is Here — Google's Open-Source LLMs Just Got a Big Upgrade

⚡ Boosting Algorithms

Boosting is a powerful ensembling technique that improves model performance by combining weak learners. In this section, we cover both the theory and practice of boosting, including hands-on guides to popular libraries:

• 📘 Concepts in Boosting Algorithms in Machine Learning

• 🛠️ Practical Guide to LightGBM

• 🛠️ Practical Guide to XGBoost

• 🛠️ Practical Guide to CatBoost

• 🛠️ Practical Guide to scikit-learn

• 🔍 LightGBM vs XGBoost vs CatBoost: A Comparison

🎰 Reinforcement Learning: Multi-Armed Bandits

An introduction to basic and contextual multi-armed bandit (MAB) problems — ideal for understanding online decision-making under uncertainty. Includes algorithm comparisons and use cases in data science and ML.

• 📘 Concepts in Multi-Armed Bandits

Quick summary

Hello 👋 In this article, I will explain how Netflix measure values of acquiring or retenting a subscriber on the following areas:

• Business scenario

• Customer lifetime value

• Markov Chain model

• Incrementality calculation

🚀 Email us @ social@verticalsolution.io if you want us to deep dive into a data science topic!

Subscribe now

Business scenario

To measure the long-term value of acquiring or retaining a subscriber in Netflix. This is important for subscription-based streaming platform like Netflix, because often one would like to know monetary benefits of certain marketing campaigns or product developments to increase acquisition of new subscribers or retention of existing subscribers.

Methodology

Need for causal model

As subscription is a self-selected behaviour, meaning that we could not control if a user subscribes or not (and hence it is not possible to conduct a AB test), one will need to rely on causal techniques to solve this problem.

Customer lifetime value (CLV)

The naive usage of CLV in this case is likely an over-estimation because a off-service subscriber could resubscribe in the future, hence the CLV from these customers are actually non-zero and have to be considered when calculating the value of acquisition or retention.

The method

Netflix data scientists come up with a nice method which combines Markov chain and causal inference to solve the problem (For an quick introduction to Markov chain, you can visit the wikipedia page here).

A Markov chain can be specified by the set of states and transition probabilities.

State definition — each subscriber (on or off-service) is represented by a state s. This state represents the number of consecutive billing cycles that the subscriber is in or not in. Each subscriber is in one of the state:

where a positive index i means that the subscriber has subscribed to the service with i consecutively billing cycle, while negative index -i means instead the subscribers has not subscribed to the service with i consecutively billing cycle. N and M are assumed to be very large.

Transition probabilities — transition probabilities between states

can also be determined by calculating empirical means or fitting a model using the historical data.

Value of a subscriber — the value function (often used in the Markov chain problem or reinforcement learning) describes the cumulative expected rewards from a state s. In this case, given a state s, the value function can be written as:

where r is the discounted factor (often used in MC or RL), I is a indicator function equal to 1 if the household is subscribed to the service at kth cycle given the current state s, and c is the price of the service. The value function can be obtained by running numerical simulations or solving the Bellman’s equation.

Calculating incremental values — After calculating the value functions or for all the states, we can then calculate the incremental value in various acquisition and retention scenarios. The value of acquiring a new subscriber is

Similarly, the value of acquiring a subscriber who has stopped the service for k months is

Summary

This post summarises the essential concepts used by Netflix data scientists on the business problem of estimating the value of customer acquisition and retention. For those who are interested more detail, you can access the paper here.

What is boosting?

In machine learning, boosting is an ensembling technique to combine a set of weak learners to form a stronger learner with smaller bias and variances. A weak (strong) learner refers to a predictor, for example a classifier or a regressor, that is weakly (strongly) correlated with the target label.

Although boosting does not require specific forms of algorithms, many boosting algorithms generally learn each weak predictor iteratively and in the end combine all weak predictors to form one strong predictor.

In mathematical form, it can be expressed as:

where

is the final predictor at n-th iteration,

is the weak predictor at k-th iteration, and

is the weight associated to the k-th weak predictor, which will be calculated differently depending on the boosting algorithm used.

In the following sections, we will explain how weak predictors and weights are determined for each boosting algorithm.

Adaboost

While there have been many developments on boosting algorithms, one of the earliest significant boosting algorithms is as known as the Adaboost algorithm.

Learning weak predictors

The core concept of Adaboost corresponds to that, at each iteration, a weak predictor is fitted to a weighted dataset, instead of the original dataset. The weight for each data point at a particular iteration is determined by its residual. Typically, the larger the residual, the larger the weight is given. This is to allow the weak predictors at later iterations to focus on correcting prediction errors of more difficult samples.

Shinkage

As iterations increase, more and more emphasis will be placed on difficult samples, hence more susceptible to the presence of outliers.

One way to ensure the stability of the algorithm against this is by employing learning rate, or also as known as shrinkage rate. The idea is that the contribution from weak learners in additional iterations will be shrinked by this rate, controlling the influences that the later-fitted learners have on the overall predictions.

The mathematical formula for this can be written as:

where n is the number of boosting iterations and w is the shrinkage rate.

Gradient boosting

Gradient boosting is a special kind of boosting algorithm that, instead of fitting at each time step a weak learner to residuals from previous iterations, it fits instead to negative gradients at that particular iteration. (Yes, even if you are training a classification model, at each iteration, except the very first iteration, the weak predictor is actually a regression model predicting negative gradients.)

In mathematical form, the update rule for the final predictor is

where similar to above,

is the final predictor (not the weak predictor!) at k-th iteration,

is the weak predictor at k-th iteration, and

is the shrinkage rate at k-th iteration.

When the weaker learner f is trained on negative gradients as target label, the

This formula looks exactly like iteration updates with gradient descent optimisation. In fact it can be shown that this iteration update is actually mathematically equivalent to gradient descent in the function space, hence the name gradient boosting.

It can be shown that this approach can generalize to various objective functions and is equivalent to performing gradient descent in the functional space. Popular open-source packages such as scikit-learn use gradient boosting in various forms.

Newton boosting

Similar to gradient boosting, at each boosting iteration, Newton boosting fits a weak predictor to a modified target label. Different from gradient boosting, instead of using negative gradients as target label, it uses negative gradients and hessian matrices to perform updates in each iteration:

Again, like gradient boosting, this update ressembles Newton’s method and corresponds to Newton's descent in functional space.

Gradient tree boosting

Actually many of the most popular open-source tree-based gradient boosting algorithms nowadays implement gradient boosting a bit differently.

In particular, like ordinary gradient boosting, at each boosting iteration, a tree is grown and added to the final predictor.

However, when growing a tree at each boosting round, the optimal split in each node is determined by maximising the loss reduction calculated by this formula:

where

where g is the gradient for data point j respectively. This formula is derived by a minimisation of a first-order Taylor approximation to the loss function.

Newton tree boosting

Similar to gradient tree boosting, one can also come up with an analogy and define a more advanced boosting algorithm which takes into the account the hessian of the loss functions.

In particular, when growing a tree at each boosting round, we use instead the second-order Taylor approximation to the loss function. All that is needed to do is to use a different loss function for node splitting.

In other words, similar to gradient tree boosting, at each node split, the optimal split is defined by the split giving the maximum loss reduction:

where

Popular open-source packages such as LightGBM and XGBoost use Newton boosting (apart from regularisation) for their main boosting method.

Summary

Understanding the fundamental concepts of boosting algorithms presented in this article is crucial for grasping the design choices behind advanced boosting algorithms like LightGBM and XGBoost.

These advanced algorithms often build upon the core principles of boosting explained here, but with additional optimisations on efficiency, scalability, and handling complex data structures. For instance, LightGBM and XGBoost leverage decision trees as weak learners and implement efficient algorithms for finding optimal splits within those trees. Their advancements lie in techniques like gradient-based decision tree learning, gradient-based sampling, and regularization to prevent overfitting.

By understanding the fundamental building blocks of boosting explored in this article, you can gain a deeper appreciation for the design philosophies behind cutting-edge boosting algorithms and how they achieve superior performance!