Benchmarking Effectiveness

This tutorial shows how to benchmark the effectiveness of a network alignment (NA) algorithm in PlanetAlign on a given dataset, which is crucial for evaluating model quality and comparing different methods.

We cover two main topics:

  1. Evaluating a model using standard metrics like Hits@K and Mean Reciprocal Rank (MRR).

  2. Visualizing the training curve to track trends over epochs (e.g., Hits@K, MRR).

These tools are essential for understanding model quality, selecting hyperparameters, and comparing competing methods.

Evaluating Effectiveness

After training a model using .train() method of the NA algorithm, you can evaluate its alignment quality using .test() method. By default, this includes:

  • Hits@K: Measures the proportion of correct alignments within the top-K predictions.

  • MRR (Mean Reciprocal Rank): Measures the average reciprocal rank of the correct alignment.

Here is an example that benchmarks the FINAL algorithm on the PhoneEmail dataset:

from PlanetAlign.datasets import PhoneEmail
from PlanetAlign.algorithms import FINAL
from PlanetAlign.logger import TrainLogger

# Load dataset
data = PhoneEmail(root='datasets/')

# Initialize and train model, add .to('cuda') if using GPU
model = FINAL()
# Initialize logger to track the training process
logger = TrainLogger(log_path='logs/', save=True)
# Train the model to align the first and second graphs
model.train(data, gids=[0, 1], logger=logger)

# Evaluate using built-in metrics
result = model.test(data, gids=[0, 1])

print("Evaluation metrics:")
for metric, value in result.items():
    print(f"{metric}: {value:.4f}")

Note

By default, .test() method reports Hits@1, Hits@10, Hits@30, Hits@50, and MRR. You can customize this using the metrics argument:

# Evaluate with Hits@1 and MRR only
result = model.test(data, gids=[0, 1], metrics=['Hits@1', 'MRR'])

Visualizing the Training Curve

To monitor model convergence and training dynamics, PlanetAlign provides a built-in utility function: plot_curve().

This function is available through the model’s logger and automatically plots key metrics (e.g., loss, MRR, Hits@K) tracked during training.

If your model logs progress via logger, you can visualize the full training history as follows:

# Visualize the training curve (MRR) of the model and save the plot
logger.plot_curve(metric='MRR', save_path='mrr_curve.png')

This will generate a line plot showing metric (MRR) over epochs, helping you identify trends like:

  • Convergence behavior

  • Optimal hyperparameter settings

Note

For visualization to work properly:

  • The model must track metrics over epochs using logger.log(epoch, metric, value).

  • plot_curve() automatically aggregates and formats the data for you.

Best Practices

  • Use multiple seeds and report mean ± std across runs for robustness.

  • Log per-epoch metrics during training for insight into convergence behavior.

Summary

  • Use test() to compute standard NA metrics like Hits@K and MRR.

  • Use built-in function of logger to visualize training progress for analysis.

  • A consistent benchmarking protocol helps compare methods fairly and reproducibly.

Next: See the “Benchmarking Scalability” tutorial to evaluate runtime and memory usage of NA algorithms.