I have pushed many companies to ensure manufacturing pilot runs take place before they go into mass production. It makes a lot of sense for electrical & mechanical products that are either totally new or new for that factory that is about to work on them.

However, I frequently get questions about what to expect, and what to test & inspect…

Manufacturing pilot runs and their objectives

The logical question people need to clarify is ‘what is the objective of each of these manufacturing pilot runs?’

There are 3 types of pilot runs. Let’s go through each of them one by one.

A. Very early pilot runs (engineering validation – often 1 to 10 samples)

These are often not called “pilots” at all. This phase is often blurred together with prototyping, and it is the work of a few engineers. 

The main objectives are:

1. Get final confirmation from users

In other words, will the product “solve their problem”, is it easy to use, and will they buy it?

2. Early confirmation of manufacturability

Are the assembly & testing operations relatively easy? And what about the manufacturing of the custom parts?

Output

The output is the approval of a final works-like, looks-like prototype (which is not yet the ‘golden sample’ because it is typically not made with production-intent processes). Good teams also keep up a list of known issues and a list of risks.

B. Early pilots (design validation – often 10 to 100 samples)

This is also done, very often, on a very small scale, by a few engineers on a bench rather than on the assembly line. And, if expected production volumes & risks are not high, that’s usually acceptable.

Here, the main objectives are:

1. Confirm that the product design is mature

If, for instance, the software integration with the hardware leaves to be desired, or if some mechanical components don’t play well together, those are red flags that need to be addressed before proceeding.

2. Confirm that the components from chosen suppliers are acceptable

Design teams often make the mistake of picking suppliers that can send a few parts quickly but are not qualified to send a mass production batch with high quality. As the team starts to work with larger numbers of parts, they need to keep track of how many are defective. In parallel, a serious job of supplier qualification is necessary on critical components.

3. Make a few finished products that are fully approved for form, fit, and function

As I wrote above, the very early pilot leads to a works-like, looks-like prototype. Here, you want a works-like, looks-like, and feel-like prototype that is made with parts that are fabricated with the same processes as mass production. That means, if there is a custom-designed plastic enclosure, you need to use parts that are made out of the final tooling or, at the very least, with some sort of soft tooling.

Note: as a reader in Linkedin pointed out, I should mention that this step relies on a preliminary step — having an engineer check the first parts off tooling. If there are issues, request Trial 2, Trial 3, etc. as required.

4. Confirm that the quality standard and the test plan are clear and workable

This is checked & confirmed at every stage, but here the standard and the test plan need to be mature. Several issues will be found in this run, and they should make it into the documentation.

5. Confirm that the product is durable & reliable

A final battery of tests gets executed at this point. (This is actually done at several stages of the process, not only in design validation, as my colleague Andrew wrote in What is the Pilot Run for Components and Products?)

Output

The output is the approval of the golden sample(s). And, as always, an update to the list of known issues and the list of risks.

C. Mid-stage & late pilots (production readiness validation – often 50 to 2000 samples)

A lot of work has been done, but is everything ready for going into mass production? If the first batch is planned to be thousands of pieces, this needs to be prepared carefully. 

The objectives are typically as follows:

1. Confirm readiness of trained people and their work instructions, of the assembly line and its fixtures (if any), of testing stations

If any of these are not ready, that’s obviously a red flag.

Always look at both the production staff and the testing/inspection staff. I wrote a simple checklist before.

2. Confirm that the proportion of defective products at the end of the line is low (below a certain percentage, for example, fewer than 2% of major issues at this stage).

Unfortunately, this is not a great measure in itself. There are often several engineers and a few inspectors watching the operations and checking work-in-process products during manufacturing pilot runs, and they tend to catch many issues that lead to reworking or discarding products. Those reworked pieces are not counted in the percentage of defective finished goods. This leads us to the next indicator…

3. Confirm the first-pass yield is above a certain percentage, for example above 80%

In assembly, this is the proportion of products that do not need any form of rework all along the line. A reworked product is more expensive to produce and it is less reliable in the long run.

In the manufacturing of custom components, or in surface treatment (e.g. paint, plating, anodizing…), the same idea. If a part needs to be touched up at the end, it leads to higher costs. And that often leads to a discussion about raising prices significantly… right when you are ramping volumes up. Just when you want to avoid it.

4. Confirm that the line can run at the expected rate, for example, “1 finished product comes off the line every 2 minutes”

You should not long look at quality risks, but also the risk of shipment delays. You don’t want to trust an over-enthusiastic contract manufacturer. If their final pilot had 20 people on the line and only 75 pcs came up in 8 hours of work, they still have a lot of work to do before that same line churns out 500 pcs a day!

Output

The output is a final confirmation that the product and the process are ready for mass production, as well as targets for mass production (run at rate, first pass yield, etc…).

And, as always, an update to the list of known issues and the list of risks.

Your comments

How do you handle manufacturing pilot runs for your products? What problems have you faced? Let me know by commenting, please.

Are you designing, or developing a new product that will be manufactured in China?

Sofeast has created An Importer’s Guide to New Product Manufacturing in China for entrepreneurs, hardware startups, and SMEs which gives you advance warning about the 3 most common pitfalls that can catch you out, and the best practices that the ‘large companies’ follow that YOU can adopt for a successful project.

It includes:

• The 3 deadly mistakes that will hurt your ability to manufacture a new product in China effectively
• Assessing if you’re China-ready
• How to define an informed strategy and a realistic plan
• How to structure your supply chain on a solid foundation
• How to set the right expectations from the start
• How to get the design and engineering right

Just hit the button below to get your copy (please note, this will direct you to my company’s website Sofeast.com):