rekall.tuner package¶
Submodules¶
Module contents¶
This module provides some classes for automatically tuning Rekall queries.
-
class
rekall.tuner.
Tuner
(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)¶ Bases:
object
Base class for all Tuners (see sub-classes for details).
Parameters: - search_space (dict) – A dictionary of parameters to search over. See note below for more details.
- eval_fn – Given a configuration, evaluate the black box function and return the score.
- maximize (bool) – Maximize the output of
eval_fn
if True, otherwise minimize it. - budget (int) – Maximum number of times to call the evaluation function.
- log (bool) – Whether to log results
- log_dir (string) – Directory to log all results to
- run_dir (string) – Directory to log results from a set of runs
- run_name (string) – Name of this run
- start_config (dict) – Some tuners ask for a starting configuration. If start_config is specified, start with this config.
- start_score (float) – If start_config is specified, you can also specify its score if you know it ahead of time.
- score_fn – Your eval function may not return exactly the value you want to optimize. This function parses the output of eval_fn to pass to the optimizer.
- score_log_fn – Your eval function may not return exactly what you want to log. This function parses the output of eval_fn to log.
- num_workers (int) – Number of workers for parallelism. See sub-classes to check whether supported. Default 1.
- show_loading (bool) – Whether to show a loading bar (not always supported).
Example:
def eval_config(params): # Run the Rekall query query_results = rekall_query( param1 = params['param1'], param2 = params['param2']) # Evaluate the results score = evaluate(query_results) return score search_space = { 'param1': [0.0, 1.0, 2.0], # discrete 'param2': { 'range': (10.0, 20.0) } # linear range } tuner = RandomTuner(search_space, eval_config, budget = 50) best_score, best_config, score_history, execution_times, total_cost = tuner.tune()
-
evaluate_configs
(configs, num_workers=None, show_loading=None)¶ Evaluate the configs.
-
log_msg
(msg)¶ Log something to the log file.
-
tune
(**kwargs)¶ Run the tuning algorithm!
-
tune_impl
(**kwargs)¶ The implementation of the tuning algorithm.
Sub-classes should implement this!
-
class
rekall.tuner.
RandomTuner
(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)¶ Bases:
rekall.tuner.tuner.Tuner
This tuner randomly searches the state space.
-
classmethod
generate_configs
(search_space, num_configs, seed=None)¶ Randomly generate
num_configs
configurations.
-
tune_impl
(**kwargs)¶ Randomly searches through the search space.
Parameters: seed (int) – Random seed to initialize things.
-
classmethod
-
class
rekall.tuner.
GridTuner
(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)¶ Bases:
rekall.tuner.tuner.Tuner
This tuner conducts a grid search over the search space.
-
classmethod
generate_configs
(search_space, budget)¶ Generate configs that cover the search space in a grid as densely as possible given the budget.
-
tune_impl
(**kwargs)¶ Performs a grid search through the search space.
-
classmethod
-
class
rekall.tuner.
CoordinateDescentTuner
(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)¶ Bases:
rekall.tuner.tuner.Tuner
This tuner performs coordinate descent over the search space.
-
line_search
(config, cur_param, epsilon, budget, cur_score=0)¶ Vary cur_param within the bounds of the search_space (holding the other parameters constant), maximizing the accuracy function.
Let X be the current param, and let F(X) represent the accuracy function. Let Y be the range of X in the search_space.
- If F(X + epsilon * Y) > F(X):
- Find the smallest positive integer l such that F(X + l * epsilon * Y) < F(X + (l - 1) * epsilon * Y), and return X + (l - 1) * epsilon * Y as the new value of the parameter
- Otherwise:
- Find the smallest positive integer l such that F(X - l * epsilon * Y) < F(X - (l - 1) * epsilon * Y), and return X - (l - 1) * epsilon * Y as the new value of the parameter
-
tune_impl
(**kwargs)¶ Start with the midpoint of the search space Then cycle through co-ordinates. For each co-ordinate:
- If the co-ordinate is discrete, try all the choices
- If the co-ordinate is numerical, run line search with alpha and a budget of 10
If all the co-ordinates stay the same, try again with alpha = alpha * decay_rate.
Parameters: - alpha – initial alpha value for line search
- decay_rate – rate to decay alpha
- init_method – How to initialize the first config.
One of
['average', 'random']
. If not specified, default to ‘average’. ‘average’ initializes the config at the average of continuous ranges, ‘random’ randomly initializes the config. If start_config was specified upon initialization, use that value always. - line_search_budget – Budget to give to line search. Must be at least 2. Defaults to 10.
- randomize_param_order – Whether to randomize the order of coordinates for coordinate descent. Defaults to False.
-
-
class
rekall.tuner.
SuccessiveHalvingTuner
(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)¶ Bases:
rekall.tuner.tuner.Tuner
This tuner does successive halving over the search space.
-
classmethod
estimate_cost
(eta, N, K, T)¶ Estimate the cost of successive halving
-
tune_impl
(**kwargs)¶ Performs successive halving - start with K random configurations, each running for T iterations of some sub-tuner. In each round, take the 1 / eta top configurations, and in the next round train for eta times more iterations.
Parameters: - eta – Halving ratio.
- N – Number of rounds.
- K – Initial number of configurations.
- T – Number of training iterations to start with.
- tuner –
Tuner
class to use for internal training rounds. - tuner_params – Optional params to pass to the internal tuner.
-
classmethod
-
class
rekall.tuner.
HyperbandTuner
(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)¶ Bases:
rekall.tuner.tuner.Tuner
This tuner performs a hyperband search over the search space.
See https://arxiv.org/abs/1603.06560.
-
classmethod
estimate_cost
(schedule)¶
-
classmethod
finite_horizon_hyperband_schedule
(max_iter, eta)¶
-
tune_impl
(**kwargs)¶ Implement hyperband search over parameter space, with a given tuner to train iterations.
See
finite_horizon_hyperband_schedule
to print out the schedule for given values of max_iter and eta.Parameters: - max_iter – Maximum number of iterations.
- eta – Proportion of configs to cut in each round of successive halving.
- tuner –
Tuner
class to use for internal training rounds. - tuner_params – Optional params to pass to the internal tuner.
-
classmethod