Treat your CESP preparation as training a diagnostic habit: after every model run, ask two questions - does the total match evidence, and do the end-use shares make engineering sense? Work through the numeric calibration exercise and the two scenarios below until you can spot a compensating error, an input that should have stayed frozen, and a schedule mistake hiding inside a plausible total.
Why a Model That Matches Bills Can Still Misrank Savings Measures
Agreement with utility data is a whole-building test. It confirms total consumption but not the end-use split, so a model can look accurate annually and still rank efficiency measures unreliably.
Whole-building calibration metrics such as NMBE and CV(RMSE) measure aggregate fit. A model can hit the annual total through compensating errors - lighting energy overstated while plug loads are understated - because the total is the only thing the utility meter can see. When you later simulate a lighting retrofit, the savings come from the lighting bucket, and an error that was invisible at the whole-building level now moves directly into the savings number.
Train yourself to ask a second question after any calibration passes: which end-use assumptions produced this total? Trace the cooling load into its components - envelope conduction, solar gains, lighting heat, people, ventilation - and compare the modeled shares against a rough order-of-magnitude estimate or any sub-metered data you can find. Only when the shares are plausible does the model become a trustworthy instrument for comparing measures, which is the central use case of simulation practice.
Computing NMBE and CV(RMSE) by Hand on a Small Monthly Dataset
NMBE shows whether a model runs consistently high or low; CV(RMSE) shows how scattered the monthly errors are. Calculating both on a six-point dataset builds the diagnostic instinct that scenario questions reward.
Use the simplified forms: NMBE equals the sum of (modeled minus measured) divided by the number of points times the mean measured value, times 100. CV(RMSE) equals the square root of the mean of the squared errors, divided by the mean measured value, times 100. Full references such as ASHRAE Guideline 14 adjust the denominator for fitted parameters and set threshold criteria, but the sign convention and the bias-versus-scatter logic are what to internalize first: a negative NMBE means the model under-predicts.
The two metrics catch different failures, so read them as a pair. A small NMBE with a large CV(RMSE) signals offsetting seasonal errors - the model is high in summer and low in winter, canceling out. A large NMBE with small scatter signals a consistent bias, typically an always-on load that is wrong by a fixed amount. If your computed values land inside commonly cited monthly calibration criteria, note that this is necessary evidence, not proof: end-use structure can still be wrong, which is exactly what the next sections train you to check.
- Exercise dataset (monthly kWh in thousands): measured 100, 120, 140, 160, 180, 200; modeled 104, 116, 145, 156, 186, 198.
- Step 1: mean of measured = 150. Step 2: errors (modeled minus measured) = +4, -4, +5, -4, +6, -2; sum = +5; NMBE = 5 / (6 x 150) x 100 = +0.56%.
- Step 3: squared errors = 16, 16, 25, 16, 36, 4; mean = 18.83; RMSE = 4.34; CV(RMSE) = 4.34 / 150 x 100 = about 2.9%.
- Expected interpretation: the errors are mixed in sign - three positive, three negative - so there is no consistent directional bias; the small net over-prediction (+0.56%) is negligible, and the low scatter means month-to-month deviations are tightly clustered around the measured values.
- Self-check rubric: (1) mean correct at 150; (2) NMBE sign is positive; (3) NMBE within 0.1 point of +0.56%; (4) CV(RMSE) within 0.3 points of 2.9%; (5) you wrote one sentence naming the error pattern as mixed-sign with negligible net bias, not a consistent over- or under-prediction. Score yourself out of 5; a 5 means ready to move on, a 3 or less means redo the arithmetic with a different dataset you invent.
Baseline Versus Proposed: The Identical-Inputs Discipline
Savings are a difference between two models, so every input the measure does not touch - weather, schedules, occupancy, non-participating systems - must stay identical across both runs.
Worked scenario 1: an office building gets a lighting retrofit. The modeler builds the proposed model and, while in there, also updates the occupancy schedules to reflect a recent shift toward hybrid work, reasoning that this makes the proposed model more realistic. The reported savings now mix the lighting measure with an unmeasured occupancy change, and because lower occupancy would itself reduce consumption, the two effects are tangled and the savings number is not attributable to the retrofit.
The better decision is to freeze every shared input and change only what the measure changes - in this case lighting power density, plus the interactive effects the simulation tool computes automatically, such as reduced lighting heat gain lowering cooling energy and slightly raising heating energy. Why it matters: in performance-oriented work the model is a measurement instrument, and any asymmetry between the runs is an uncontrolled variable. Before trusting any savings figure, list the inputs that changed and verify each one maps to a specific measure.
Choosing Weather Files and Simulation Periods That Match the Question
Typical-year weather suits long-term average savings estimates; actual-year weather suits reconciling a specific historical year; design-day runs test peak capacity, not annual energy. Match the file to the claim being made.
Comparing a model against the bills of a particular year requires the actual weather of that year, because an unusually mild or severe season shifts consumption for reasons that have nothing to do with the building. Estimating the value of a measure over its lifetime calls for a typical meteorological year or a multi-year average, since the business question is the expected performance, not one year's luck. A frequent trap is calibrating to actual-year bills and then quoting typical-year savings as if they were validated against the calibration, when the two runs answer different questions.
Design-day simulations sit in a separate category: they stress the building under near-worst conditions to check equipment capacity and size, and their results should never be read as annual energy use. Also check the calendar logic of whatever file you use - weekday, weekend, and holiday patterns differ between years and between typical-year files, and a schedule applied on the wrong days distorts both energy totals and demand shapes. Stating which weather file, which version, and which calendar you used is part of the documentation habit covered later.
End-Use Sanity Checks That Catch Structural Errors
Before trusting measure results, compare modeled end-use shares against order-of-magnitude engineering judgment and any sub-metered or spot data. A plausible total built on implausible shares signals structural error.
Worked scenario 2: an office model calibrates well against annual bills, and its end-use table shows cooling at roughly 55% of consumption - far above what the climate, envelope, and internal gains should produce. Tracing the cooling end-use back through its drivers reveals the lighting schedule runs 24 hours a day, so the model carries lighting heat around the clock and dumps it into the cooling load. The modeler had accepted the calibration because totals matched, then used the model to evaluate a chiller replacement and obtained inflated savings, since the chiller was serving a phantom load.
The better decision is to stop at the implausible share, fix the lighting schedule to match actual operating hours, recalibrate, and rerun before evaluating any measure. Why it matters: the ranking of measures can flip once the structure is right - a chiller replacement that looked dominant may drop below scheduling and controls measures once the phantom load disappears. Build the reflex of reading the end-use table first, the savings table second.
The table below summarizes which evidence validates which claims, so you can pick the right check for the right question.
| Evidence source | What it can validate | What it cannot validate |
|---|---|---|
| Monthly utility bills | Whole-building total consumption and seasonal shape | Individual end-use shares; hourly load shape |
| Hourly interval data | Load shape and timing of peaks | End-use attribution without sub-metering |
| Sub-metered circuits or end-use meters | One specific end-use's magnitude and schedule | Other end-uses; interactive effects between systems |
| Short-term spot measurements (light levels, flows, temperatures) | A single assumption such as installed power or setpoint | Annual behavior; diversity across the building |
| Design-day run | Peak capacity and equipment sizing | Annual energy consumption |
Documentation That Makes a Model Auditable and Defensible
A simulation is only as usable as its record: weather file and version, schedule sources, envelope inputs, calibration report, software version, and run history let a reviewer reproduce every number.
Log each input with its source - measured, assumed, from drawings, or from a default - because reviewers and downstream users need to know which numbers carry evidence and which carry judgment. Keep one controlled model tree with dated versions and a written change log for each run, rather than scattered what-if copies; undocumented copies are a classic source of contradictions when two team members quote different savings from what turns out to be two different files.
Documentation also carries the professional dimension of the work: state the assumptions that most influence results, be explicit about uncertainty, and say plainly when a question falls outside what the model was calibrated to represent instead of presenting an exact figure. Presenting a savings estimate with its validation basis and its limits is more defensible - and more ethical - than presenting a bare number, and building this habit during study makes it automatic under exam-style scenarios and in practice alike.
A Six-Week Preparation Sequence and Readiness Checks
Sequence your preparation around doing, not rereading: alternate concept study with spreadsheet exercises, add one paper scenario per week, and close with timed mixed review across all the exam's stated domains.
Weeks 1-2: core building energy concepts - envelope heat transfer, internal gains, HVAC basics - while building a simple monthly spreadsheet end-use model and computing NMBE and CV(RMSE) by hand on datasets you invent. Weeks 3-4: calibration practice on the spreadsheet model, then the baseline-versus-proposed discipline using a paper scenario, writing out which inputs changed and why. Weeks 5-6: weather-file logic, documentation practice, professional standards, and timed mixed sets drawn from the exam's topic areas: energy concepts, simulation assessment and interpretation, applied decision-making, methods and documentation, ethics and standards, and case analysis. Compress or stretch the six weeks to fit your available time; the order matters more than the duration. For administrative matters - eligibility, required training, application, and scheduling - consult the issuer directly at the AEE certifications page linked below, and note that AEE describes an approved training step for its certification process.
Use the readiness checks below as your finish line, and treat a self-check score as a learning milestone, not a prediction of any passing outcome.
- You can compute NMBE and CV(RMSE) on a fresh twelve-point dataset in under ten minutes, with correct signs and a one-sentence interpretation.
- Given any savings report, you can list the inputs that must be identical across baseline and proposed runs and identify a mismatch if one exists.
- You can read a modeled end-use table and name at least two checks you would run before trusting it.
- You can explain in two sentences each when to use typical-year weather, actual-year weather, and design-day runs.
- You can outline a documentation package that another modeler could reproduce from without asking you a single question.
References and further reading
Use these references to explore the concepts and check the latest information from the relevant organizations.