> For the complete documentation index, see [llms.txt](https://docs.dinmo.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.dinmo.io/intelligence/product-recommendations/understand-performance.md). # Understand performance Read technical delivery, offline relevance, and business impact separately. A successful run does not prove that recommendations are relevant or that a campaign will generate incremental revenue. ## Check delivery first Confirm that the expected outputs were published recently, then inspect their population and coverage: * **Source coverage:** the proportion of eligible sources receiving at least the configured minimum number of results. * **Complete coverage:** the proportion receiving K results. * **Coverage fill:** how much of the output was completed using general demand rather than personalized relationship evidence. Always read the denominator. An output can cover all eligible users while excluding part of the originally requested population. Row counts are not unique customer counts. For allocation, compare assigned unique users with the requested audience size, and inspect the achieved target shares. ## Read the training evaluation Offline evaluation uses historical decision dates. At each cutoff, the model learns from earlier interactions and is evaluated against objective events in the following time window. The final model is then trained on the admitted training history. | Metric | Interpretation | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | Hit rate at K | Share of evaluable user-date observations with at least one relevant item in the top K. A 12% hit rate is not a 12% conversion probability. | | NDCG at K | Ranking quality that rewards relevant items near the top. | | Average first-hit rank | Average position of the first relevant item, only among observations with a hit. Lower is better within that population. | | Random baseline | A reference using eligible candidates. Beating it is a sanity check, not proof of improvement over your existing campaigns. | First-hit rank can improve while hit rate falls: the remaining successful recommendations are better placed, but fewer observations have any hit. Read both metrics together. ## Understand the evaluation population An observation is a user at a historical cutoff, not necessarily a distinct customer. Evaluation samples at most 5,000 measurable observations per cutoff. This bounds evaluation work; it does not impose the same limit on training users or events. For items without a known end date, evaluation uses recent activity or a recent opening date in the seven days preceding the cutoff to narrow scoring. This is an evaluation fallback, not a rule silently applied to commercial recommendation segments. Within the configured evaluation universe, the reference eligible universe remains broader than this time-based scoring shortlist: relevant outcomes outside the shortlist still count as misses. Historical dates prevent learning from future events, but snapshot item attributes are not necessarily a complete record of what was known at every historical cutoff. Treat that as a limitation when interpreting results. An insufficient-evidence result means the available observations do not support the requested measurement. It is not the same as a measured zero hit rate. When too few historical examples are available, DinMo displays an insufficient-evidence message instead of presenting invented or zero-valued quality metrics. Compatible predictions can still use the trained model. Use **Available models** to identify models that predictions can currently reuse, and **Training history** to review individual attempts. A failed retraining attempt does not replace the previous compatible model. When evaluation is limited to recommendable products, metrics use only products in the selected segment that existed at each cutoff. User-cutoff observations whose relevant outcomes are all outside that segment are excluded rather than counted as misses. The quality card identifies the selected segment. Compare runs only when they use the same evaluation scope. ## Compare like with like Compare metrics only when the objective event, K, horizon, historical cutoffs, eligibility rules, and sampling method match. A seven-day test and a thirty-day test answer different questions. Training evaluation measures user-to-item relevance. It does not directly validate related-item quality, overlapping audiences, vendor/category outputs, or an allocation's commercial outcome. Evaluation also separates relationship evidence from coverage fill: a full published list is not proof of a fully personalized list. ## Validate business impact Inspect representative warm and cold users, recently opened items, and recurring purchases. Verify that outputs respect the campaign's rules before activation. Then compare against a business baseline with a controlled campaign or A/B test. Measure the actual objective—incremental revenue, contributions, engagement, or another outcome—alongside reach, fatigue, and unsubscribes. Affinity scores and offline metrics alone cannot prove commercial lift. For row-level scores and activation checks, see [Use results and best practices](/intelligence/product-recommendations/use-results-and-best-practices.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.dinmo.io/intelligence/product-recommendations/understand-performance.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.