probnmn.evaluators.question_coding_evaluator¶
-
class
probnmn.evaluators.question_coding_evaluator.
QuestionCodingEvaluator
(config: probnmn.config.Config, models: Dict[str, Type[torch.nn.modules.module.Module]], gpu_ids: List[int] = [0], cpu_workers: int = 0)[source]¶ Bases:
probnmn.evaluators._evaluator._Evaluator
Performs evaluation for
question_coding
phase, using batches of evaluation examples fromQuestionCodingDataset
.- Parameters
- config: Config
A
Config
object with all the relevant configuration parameters.- models: Dict[str, Type[nn.Module]]
All the models which interact with each other for evaluation. This should come from
QuestionCodingTrainer
.- gpu_ids: List[int], optional (default=[0])
List of GPU IDs to use or evaluation,
[-1]
- use CPU.- cpu_workers: int, optional (default = 0)
Number of CPU workers to use for fetching batch examples in dataloader.
Examples
To evaluate a pre-trained checkpoint:
>>> config = Config("config.yaml") # PHASE must be "question_coding" >>> trainer = QuestionCodingTrainer(config, serialization_dir="/tmp") >>> trainer.load_checkpoint("/path/to/question_coding_checkpoint.pth") >>> evaluator = QuestionCodingEvaluator(config, trainer.models) >>> eval_metrics = evaluator.evaluate(num_batches=50)
-
evaluate
(self, num_batches:Union[int, NoneType]=None)[source]¶ Perform evaluation using first
num_batches
of dataloader and return all evaluation metrics from the models.- Parameters
- num_batches: int, optional (default=None)
Number of batches to use from dataloader. If
None
, use all batches.
- Returns
- Dict[str, Any]
Final evaluation metrics for all the models.
-
_do_iteration
(self, batch:Dict[str, Any]) → Dict[str, Any][source]¶ Perform one iteration, given a batch. Take a forward pass to accumulate metrics in
ProgramGenerator
andQuestionReconstructor
.- Parameters
- batch: Dict[str, Any]
A batch of evaluation examples sampled from dataloader.
- Returns
- Dict[str, Any]
A dictionary containing model predictions and/or batch validation losses of
ProgramGenerator
andQuestionReconstructor
. Nested dict structure:{ "program_generator": {"predictions", "loss"} "question_reconstructor": {"predictions", "loss"} }