rekall.tuner package

Module contents

This module provides some classes for automatically tuning Rekall queries.

class rekall.tuner.Tuner(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)

Bases: object

Base class for all Tuners (see sub-classes for details).

Parameters:
  • search_space (dict) – A dictionary of parameters to search over. See note below for more details.
  • eval_fn – Given a configuration, evaluate the black box function and return the score.
  • maximize (bool) – Maximize the output of eval_fn if True, otherwise minimize it.
  • budget (int) – Maximum number of times to call the evaluation function.
  • log (bool) – Whether to log results
  • log_dir (string) – Directory to log all results to
  • run_dir (string) – Directory to log results from a set of runs
  • run_name (string) – Name of this run
  • start_config (dict) – Some tuners ask for a starting configuration. If start_config is specified, start with this config.
  • start_score (float) – If start_config is specified, you can also specify its score if you know it ahead of time.
  • score_fn – Your eval function may not return exactly the value you want to optimize. This function parses the output of eval_fn to pass to the optimizer.
  • score_log_fn – Your eval function may not return exactly what you want to log. This function parses the output of eval_fn to log.
  • num_workers (int) – Number of workers for parallelism. See sub-classes to check whether supported. Default 1.
  • show_loading (bool) – Whether to show a loading bar (not always supported).

Example:

def eval_config(params):
    # Run the Rekall query
    query_results = rekall_query(
        param1 = params['param1'],
        param2 = params['param2'])

    # Evaluate the results
    score = evaluate(query_results)

    return score

search_space = {
    'param1': [0.0, 1.0, 2.0],          # discrete
    'param2': { 'range': (10.0, 20.0) } # linear range
}

tuner = RandomTuner(search_space, eval_config, budget = 50)

best_score, best_config, score_history, execution_times, total_cost = tuner.tune()
evaluate_configs(configs, num_workers=None, show_loading=None)

Evaluate the configs.

log_msg(msg)

Log something to the log file.

tune(**kwargs)

Run the tuning algorithm!

tune_impl(**kwargs)

The implementation of the tuning algorithm.

Sub-classes should implement this!

class rekall.tuner.RandomTuner(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)

Bases: rekall.tuner.tuner.Tuner

This tuner randomly searches the state space.

classmethod generate_configs(search_space, num_configs, seed=None)

Randomly generate num_configs configurations.

tune_impl(**kwargs)

Randomly searches through the search space.

Parameters:seed (int) – Random seed to initialize things.
class rekall.tuner.GridTuner(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)

Bases: rekall.tuner.tuner.Tuner

This tuner conducts a grid search over the search space.

classmethod generate_configs(search_space, budget)

Generate configs that cover the search space in a grid as densely as possible given the budget.

tune_impl(**kwargs)

Performs a grid search through the search space.

class rekall.tuner.CoordinateDescentTuner(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)

Bases: rekall.tuner.tuner.Tuner

This tuner performs coordinate descent over the search space.

Vary cur_param within the bounds of the search_space (holding the other parameters constant), maximizing the accuracy function.

Let X be the current param, and let F(X) represent the accuracy function. Let Y be the range of X in the search_space.

If F(X + epsilon * Y) > F(X):
  • Find the smallest positive integer l such that F(X + l * epsilon * Y) < F(X + (l - 1) * epsilon * Y), and return X + (l - 1) * epsilon * Y as the new value of the parameter
Otherwise:
  • Find the smallest positive integer l such that F(X - l * epsilon * Y) < F(X - (l - 1) * epsilon * Y), and return X - (l - 1) * epsilon * Y as the new value of the parameter
tune_impl(**kwargs)

Start with the midpoint of the search space Then cycle through co-ordinates. For each co-ordinate:

  • If the co-ordinate is discrete, try all the choices
  • If the co-ordinate is numerical, run line search with alpha and a budget of 10

If all the co-ordinates stay the same, try again with alpha = alpha * decay_rate.

Parameters:
  • alpha – initial alpha value for line search
  • decay_rate – rate to decay alpha
  • init_method – How to initialize the first config. One of ['average', 'random']. If not specified, default to ‘average’. ‘average’ initializes the config at the average of continuous ranges, ‘random’ randomly initializes the config. If start_config was specified upon initialization, use that value always.
  • line_search_budget – Budget to give to line search. Must be at least 2. Defaults to 10.
  • randomize_param_order – Whether to randomize the order of coordinates for coordinate descent. Defaults to False.
class rekall.tuner.SuccessiveHalvingTuner(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)

Bases: rekall.tuner.tuner.Tuner

This tuner does successive halving over the search space.

classmethod estimate_cost(eta, N, K, T)

Estimate the cost of successive halving

tune_impl(**kwargs)

Performs successive halving - start with K random configurations, each running for T iterations of some sub-tuner. In each round, take the 1 / eta top configurations, and in the next round train for eta times more iterations.

Parameters:
  • eta – Halving ratio.
  • N – Number of rounds.
  • K – Initial number of configurations.
  • T – Number of training iterations to start with.
  • tunerTuner class to use for internal training rounds.
  • tuner_params – Optional params to pass to the internal tuner.
class rekall.tuner.HyperbandTuner(search_space, eval_fn, maximize=True, budget=500, log=False, log_dir=None, run_dir=None, run_name=None, start_config=None, start_score=None, score_fn=<function Tuner.<lambda>>, score_log_fn=<function Tuner.<lambda>>, num_workers=1, show_loading=True)

Bases: rekall.tuner.tuner.Tuner

This tuner performs a hyperband search over the search space.

See https://arxiv.org/abs/1603.06560.

classmethod estimate_cost(schedule)
classmethod finite_horizon_hyperband_schedule(max_iter, eta)
tune_impl(**kwargs)

Implement hyperband search over parameter space, with a given tuner to train iterations.

See finite_horizon_hyperband_schedule to print out the schedule for given values of max_iter and eta.

Parameters:
  • max_iter – Maximum number of iterations.
  • eta – Proportion of configs to cut in each round of successive halving.
  • tunerTuner class to use for internal training rounds.
  • tuner_params – Optional params to pass to the internal tuner.