Service 03 — ¥29,000 JPY · 3 weeks
A second look at your pilot — from outside the supplier relationship
A favourable pilot report from a supplier answers certain questions. What it rarely answers is whether the pilot conditions resembled ordinary operation, or what the cost would be when the system is running at full scale.
A pilot report is one view of what happened — this engagement provides another.
What this engagement delivers
An independent assessment of what your pilot actually showed
After three weeks, you hold three things: an assessment of the measurement method used in the pilot, figures recalculated at operational scale, and a recommendation — which may be to discontinue.
That last point matters. We consider a recommendation to discontinue a complete and acceptable outcome, and we write it in the same plain terms as any other finding.
Assessment of the measurement method
We examine how performance was measured during the pilot — what was counted, what was excluded, and whether the method would produce the same results under different conditions.
Figures recalculated at operational scale
Pilot conditions are rarely identical to full-scale operation. We recalculate the figures using assumptions that reflect how the system would actually run — volume, load, and the edge cases that the pilot did not encounter.
A recommendation that may be to discontinue
The engagement concludes with a written recommendation. Where the recalculated figures or the assessment of pilot conditions do not support wider deployment, that is stated plainly.
The situation this addresses is one where the pressure to proceed often outpaces the basis for it.
The situation this is suited to
A favourable pilot report, and the question of what to do with it
The report
The supplier's pilot shows promising figures
The report presents numbers that make the case for wider deployment. The question that follows — whether those numbers would hold under ordinary operating conditions — is one the supplier has a commercial interest in not examining too closely.
The gap
Pilot conditions and operational conditions are rarely the same
Pilots are often run on a selected subset of traffic, with additional attention from supplier staff, and over a period that avoids the seasonal or volume peaks that full-scale operation would encounter regularly.
The cost
Full-scale costs are often not shown in pilot reports
Running costs at scale — licensing, maintenance, staff time for oversight, and the load on existing systems — are frequently not addressed in pilot reports, or are addressed in terms that make them appear smaller than they will be in practice.
The approach is to examine what the pilot actually measured — and what it did not.
How the evaluation works
We examine what the pilot measured against what full-scale operation requires
The evaluation sits between the supplier's pilot report and your organisation's deployment decision. It examines the gap between those two points — looking at measurement method, representativeness of pilot conditions, and cost at scale.
Join diagram — Pilot Evaluation Engagement
Figures and claims
What was counted
Independent review point
Real conditions
The orange node marks the independent review point — where the method is examined and the figures are recalculated before a recommendation is formed.
What the evaluation examines
- —The measurement method used in the pilot — what was included and what was excluded
- —Whether pilot conditions resembled ordinary operation in volume, scope, and edge cases
- —Figures recalculated at full operational scale using realistic assumptions
- —Running cost at scale — including licensing, maintenance, and oversight requirements
- —Whether the result supports wider deployment, or whether discontinuing is the more defensible conclusion
What you bring to the engagement
The evaluation works from the pilot documentation you hold — the report, the measurement data, and any agreement terms or running cost figures provided by the supplier. You do not need to produce additional data; the evaluation examines what already exists.
We may ask questions of the staff who were involved in the pilot period, to understand how closely pilot conditions matched the ordinary operating environment.
Three weeks is sufficient for a thorough evaluation — it is also deliberately bounded.
What three weeks looks like
Document review, recalculation, and written output
Week 1
Document review and method assessment
We review the pilot documentation — the report, measurement data, and supplier-provided cost figures. The measurement method is assessed: what it counted, how it counted it, and where the approach introduces optimistic bias.
Week 2
Scale recalculation and cost modelling
The pilot figures are recalculated using operational-scale assumptions — realistic volume, load variation, and the edge cases that full deployment would introduce. Running cost is modelled separately from the supplier's projection.
Week 3
Written output and recommendation
Findings are compiled into a written document covering the method assessment, recalculated figures, and a recommendation. The recommendation may support wider deployment, recommend further investigation, or recommend discontinuing.
The investment is the smallest of the three services — the scope is bounded by what a pilot already produced.
Investment
Three weeks, one price, one written output
Pilot Evaluation Engagement
¥29,000 JPY
Three-week engagement · Single payment
The fee covers the full three-week evaluation — document review, method assessment, scale recalculation, cost modelling, and the written output with recommendation. There are no add-ons and no expectation of further work.
A recommendation to discontinue is covered by the same fee as any other conclusion. We do not adjust the output based on what outcome would be commercially convenient for either party.
What is included
- —Assessment of the measurement method used in the pilot
- —Review of whether pilot conditions resembled ordinary operation
- —Figures recalculated at operational scale
- —Running cost modelled at full scale, independently of supplier projections
- —Written recommendation — which may be to discontinue
- —Written output delivered as a document your team can use independently
The evaluation methodology is designed to surface the specific ways pilot reports tend to overstate results.
How the work is conducted
Three patterns that produce optimistic pilot figures — and how we address each
Selected traffic
Pilots are often run on the cleaner, more representative subset of incoming cases — excluding the ambiguous, unusual, or high-stakes examples that full operation would include. We examine what was excluded and model its effect on performance.
Supplier attention during the pilot period
Suppliers typically assign additional staff to pilot engagements. The results reflect that attention — which is not a resource that continues at full-scale deployment. We note where this effect is likely to be present.
Costs not counted in the pilot
Ongoing maintenance, staff time for oversight, and the incremental load placed on existing systems are frequently absent from pilot cost figures. These are recalculated separately and included in the full-scale assessment.
Data handling — this engagement
What to expect as outcomes
A clear account of what the pilot actually showed
The method assessment gives you a documented understanding of what the pilot's figures are based on — one that is independent of the supplier's account of them.
Recalculated figures at operational scale
The deployment decision can be made against numbers that reflect how the system would actually run — not how it performed under pilot conditions. These figures may be higher or lower than what the report shows.
A written recommendation you can act on
The recommendation is written in plain terms and does not hedge toward a particular outcome. If the figures support deployment, that is stated. If they do not, that is stated with equal clarity.
A short conversation is enough to establish whether this engagement is suited to what you are holding.
Our commitment
What we stand behind
No commercial relationship with suppliers
Orb Lab Mesh has no commercial relationship with the suppliers whose pilots we evaluate. The assessment is not influenced by any party other than the organisation that engaged us.
Written output, always
The evaluation will produce a written document — not a verbal summary or a slide presentation. The findings and recommendation stay with you in a form that can be shared, referenced, and used independently.
An initial conversation at no charge
If you would like to discuss your pilot situation and whether this engagement would be suited to it, the initial conversation carries no charge and no expectation of proceeding.
Getting started requires only a message — the next steps are straightforward from there.
How to begin
From your pilot documentation to a written recommendation
Step 01
Send a message
Use the contact form or email us at info@orb-labmesh.com. A brief description of the pilot — what the supplier has provided, what decision you are facing, and when you would need a recommendation — is enough to start.
Step 02
Initial conversation
We will respond within two business days to schedule a short conversation. This is where we establish what documentation is available and whether the evaluation would produce something useful for your situation.
Step 03
Evaluation begins
If both sides agree to proceed, you provide the pilot documentation and we agree on a start date. The three-week period begins from the point at which the documentation is in hand.
Two other services may suit a different stage or a different aspect of the same decision.
Other services
If a different engagement fits your situation
¥40,000 JPY — 5 weeks
Integration Architecture Review
A structured assessment of how a newly introduced tool would sit alongside your existing systems — covering data flow, permission boundaries, points of failure, and the maintenance each connection would require.
¥35,000 JPY — 7 weeks
Email and Correspondence Handling
Setting up assisted handling of high-volume incoming correspondence — classification, routing, and reply drafting — with a human review step retained before anything leaves the organisation.
Pilot Evaluation Engagement
An independent view of what your pilot actually showed
If you are holding a supplier pilot report and would like an independent assessment of what it means for a deployment decision, a conversation is a reasonable place to begin.
Start a conversation