AI RESEARCH ANALYSIS

A Structured Taxonomy for Multi-Fidelity HPO

We propose a comprehensive taxonomy for multi-fidelity HPO algorithms, structured around key decision points:

• Single-Fidelity vs. Multi-Fidelity: Distinguishes traditional HPO from approaches using partial assessments.
• Budget as Decision Variable: Whether the optimizer actively decides resource allocation (budget-dependent) or follows predefined schemes (budget-independent).
• Action on Budget Exhaustion: How methods handle configurations when the budget is spent (performance prediction vs. multi-armed bandit).
• Budget Inclusion in Method: For budget-dependent approaches, how fidelity is integrated (e.g., Multi-Fidelity Bayesian Optimization vs. other methods).
• Prediction Method: For performance prediction, whether curve-fitting or black-box models are used.
• Budget Calculation: For multi-armed bandit, whether a single fixed budget or several options (e.g., Hyperband idea) are used.

Performance Prediction Approaches

Key Point: These models use lower fidelity evaluations to predict the possible final performance of a configuration, guiding the HPO process. They assess less resource-intensive (and less reliable) partial evaluations to estimate the complete model evaluation.

Possible Split:

• Curve-fitting approaches: Parametrically model the learning curve, extrapolating from initial parts of the curve to predict final performance.
• Black-box prediction approaches: Directly predict a final value using an ML model (e.g., regression) fed with partial curve data and configuration features, without predefined functional forms.

Strengths: Aims to enhance decision reliability by thoroughly understanding potential final value. Weaknesses: Can incur high computational cost if prediction model training is expensive; lacks full theoretical guarantees for learning curve prediction; conceptually more complex.

Multi-Armed Bandit Algorithms

Key Point: This strategy partially evaluates resource-constrained configurations, compares results, and allocates more resources to the most promising ones, discarding the rest. It progresses across multiple rounds, stopping less promising candidates early.

Possible Split:

• Fixed Budget: Predetermined number of configurations and resources per step (e.g., Successive Halving).
• Based on HB Idea: Dynamic allocation of resources across iterations with varying configurations and budgets (e.g., Hyperband).

Strengths: Generally do not require extra computational cost for future predictions; conceptually simple; suitable for asynchronous study. Weaknesses: Often relies on random search for sampling new configurations, making robust decisions difficult; high rate of pruning promising early-stage configurations; sensitive to initial budget decisions.

Key Experimental Outcomes & Learnings

Our extensive experimental study across classification and regression tasks using various DL models reveals critical insights into the practical effectiveness of multi-fidelity algorithms. We compared representative algorithms from each taxonomic group: Performance Prediction (Learning Curve, FABOLAS), Multi-Armed Bandit (Hyperband, BOHB), and Budget Dependent (BOIL).

Key Findings: Multi-armed bandit algorithms, particularly BOHB and Hyperband, consistently demonstrated superior performance and significantly lower computational costs across most scenarios. Performance prediction methods showed varying reliability, while budget-dependent approaches generally incurred higher computational costs due to increased complexity in their search space.

Lessons Learned: Reducing fidelity can lead to worse performance if not managed carefully. The search algorithm is critical. Simple, robust methods like Hyperband and BOHB, which effectively prune unpromising configurations, often outperform more complex prediction-based or budget-dependent strategies. Avoiding computationally expensive prediction tasks and premature discarding of good configurations are crucial for optimal HPO.