Before training a model, design the evaluation

It is tempting to start a machine-learning project by selecting an architecture. A more useful first step is to define what a convincing result would look like.

An evaluation plan connects the model to the question being asked. Without that plan, it is easy to spend time improving a score that does not describe the intended task.

Write down the prediction question

Describe the input, the target, and when the prediction would be made. For a forecasting problem, ask what information would actually be available at that moment. For image classification, consider where new images will come from and how similar they may be to training examples.

This description helps determine how the data should be separated. A convenient random split is not automatically a good representation of future use.

Keep evaluation information out of training

The test set should represent an independent check, not a source of repeated design feedback. Decisions learned from data, including preprocessing, need to respect the training boundary.

Near-duplicate samples and related observations deserve attention too. Two files can be different while still carrying almost the same information. Record how related samples were grouped and why that grouping fits the research question.

Establish a baseline

Choose a simple comparison that is appropriate to the task. Then evaluate it using the same split and metric as the more complex approach. A baseline gives the final score context and can expose problems in the experimental setup early.

Keep a compact experiment record:

• The dataset version and inclusion rules.
• The split strategy and any grouping constraints.
• Preprocessing choices and training configuration.
• The baseline and evaluation metrics.
• Observed errors and known limitations.

Look beyond one score

Inspect examples where the model fails. Are errors concentrated in a particular condition, device, location, or class? An aggregate result can hide patterns that matter to the intended application.

Describe those limitations alongside the result. Clear boundaries make a study easier to interpret and give the next experiment a concrete starting point. A carefully designed evaluation is part of the contribution, even when the model itself is uncomplicated.