Hopefully, you already appreciate the importance of doing a pilot run (we already wrote about that here) to test all kinds of things before mass production starts, but here’s a quick recap about why it’s an essential part of the new product introduction process and some of the key points you need to plan for ahead of performing the pilot in order to get a successful outcome.
Prefer listening to reading?
What is a pilot run?
A pilot run is where you do a test run of manufacturing the product before mass production begins. The goal is to find and fix issues that would otherwise be a big problem if replicated in, say, thousands of units being mass-produced. It allows you to perfect the production process, debug issues, calibrate the equipment, etc. Not doing a pilot run means you’re going into mass production blind, and any number of damaging problems could occur which I’ll come on to next. (01:10)
Important ways to get the best out of pilot runs.
Before you’re ready to get started, consider these points first:
- Plan your pilot run ahead of time – done in advance a pilot run will help you decide a number of things you need for manufacturing, such as what equipment, how much space, how many staff, etc.
- Make sure all teams are trained – do your different teams, such as production, assembly, QC, warehousing and logistics, etc, have training on the equipment, the ability to assess what is good and bad, testing procedures, jigs and fixtures, etc before mass production starts? The pilot run is a great time to do training and work out where more is required.
- Make use of the lag between hardware and software readiness – software usually lags behind hardware by some weeks, so once the hardware design is locked the hardware team can be moved to focus on the pilot run while the software is being finished and can help arrange the production line for it to take place on. Once the software is complete, a test with it on the hardware will be needed, but this is probably not as involved as the pilot run.
- Be sure that equipment is ready – manufacturing requires multiple teams to do multiple tasks and is complex. Sometimes it’s easy to forget to calibrate the equipment, but this could lead to product reliability failures in the field or defects in thousands of pieces. For example, you would need to make sure that the solder temperature is correct and consistent when producing PCBAs.
- Prepare the processes and make them ready – you need to select the most efficient process with the best yield, so you need to examine the process on paper in advance and devise improvements which could mean using automation, adding staff in an area where there might be bottlenecks, etc. Process engineering in this way is one of the most critical tasks for successful mass production. (02:46)
Common problems that the pilot run will uncover that you will want to avoid.
Pilot runs are beneficial because they allow you to find issues before you start producing thousands of products which could all otherwise end up as scrap. Logically, what business can risk that happening?
By doing a small run of maybe even less than 100 pieces, you will find all kinds of problems like:
- Issues with pick and place machines
- Pasting and stencil issues
- Parts missing or not fitting
- Soldering issues, such as it being out of place or the wrong temperature
- Testing stations not where they need to be on the line (for some complex products, they needed to be tested at different points during production before the entire unit is assembled because once assembled the modules and/or components are so tightly packed inside the housing it is impossible)
- Automation and jigs are in the wrong position or need to be added to the line
- …and more (10:12)
How it helps to control processes.
Typically the factory should be using a process control plan, statistical process control for tracking each process, control charts that tell you if processes are in or out of control and their timings, and monitoring yields on a regular basis. The pilot run lets you test all of these in action and make adjustments where needed.
…and the testing of automation and jigs
Automation and jigs also need to be planned as these will increase the consistency of work coming off the production line, as without them operators’ output might vary depending on if they’re tired, sick, in a bad mood, etc. Jigs are tools that will assist operators to do a process correctly, for example showing a green light if a component has been attached properly, but red if it is not and needs to be done again. Automation is important for many reasons, for example, testing RF can be done very quickly by automatically moving products into the testing chamber where the equipment can check the RF emissions in seconds and then move on to the next product. Once again, the pilot run allows the testing of jigs and automation before mass production starts. (13:17)
5 steps to plan for before you start your pilot run.
- Consider the scope of the pilot run. For example, you may run a pilot for different reasons and the scope may be quite narrow or broad, perhaps just to test a component or PCB, supplier qualification, software testing, testing a new piece of manufacturing equipment, or testing the full production process before mass production.
- What are the main objectives? For example, if you’re doing a pilot run to test components, your objectives might be to know if the component is good enough quality, if the pick and place machines can place it properly, etc. You might have to do this many times if you have multiple components in the product.
- Who will perform it? You’ll make a pilot run team, so who is in it and what are their responsibilities? The person in charge needs to be knowledgeable about the product, design, manufacturing, components, etc, depending on what is being tested. Ideally, the team will be led by a senior staff member, such as a high-level engineer or project manager who understands the process very clearly from end to end and has the authority and experience to coordinate the different team members.
- Do you have the plan to carry it out? The pilot run plan may include schedules, team, equipment, and components involved, as well as details for second suppliers in case the primary supplier runs out of stock, and anything else important, too.
- How will you document and evaluate the pilot run? In order to assess the pilot run’s effectiveness, you need to evaluate every step of the process and document any issues found and how they were resolved. The documentation may include lessons learned analysis, corrective action plans, and more, and should be retained so it can be used to run future pilot runs more easily. The documentation is also living, as it will be updated over time as more pilot runs are done and improvements to processes are made. (17:50)
How does the cost of doing pilot runs compare with not doing them?
Pilot runs vary considerably depending on the products, goals, etc, outlined here, so it’s hard to give a general ‘cost’ for them. However, if you do not do pilot runs you risk schedule and funding overruns during mass production because you’ll run into so many issues (that could have been found and fixed during the pilot) that you’ll need to spend time and money on activities like reworking defective products, overtime pay for operators, etc. There are also costs associated with brand damage attributed to unreliable products, customer warranty returns, and lawsuits or product recalls due to unsafe products, for example.
The pilot run should not even be expensive to set up and run, especially if the manufacturer has already made similar products on a similar production line which can simply be adjusted to the new requirements by adding relevant machines, jigs, automation, operators, additional lines, etc, where needed for the pilot. Not doing a pilot could easily cost 10x more in terms of negative effects. (25:50)
Do Asian manufacturers have the habit of doing pilot runs?
It’s not uncommon for manufacturers to skip the pilot run and rush into mass production as they’re incentivised to ship your products as soon as possible in order to get their payment. For the reasons mentioned here, that’s not good, but less-experienced buyers often fall into the trap of trusting their supplier too much because ‘they know what they’re doing.’ This can lead to important elements being skipped to save them tie and money, such as quality and reliability testing, pilot runs, using adequate packaging that actually protects the products during shipping, etc. (27:48)
Conclusion.
A successful pilot run is very beneficial, resulting in smoother mass production, fewer risks (those risks that remain will be known and manageable), and better yields. (29:16)
Excellent advice, as always, thanks!
Let me offer one specific addition/ caution. For the most effective pilot, you want just about everything “production intent”. Parts made from production tooling, in the regular production environment; all assembly done in the intended production facility by regular production personnel, and so on. Quite likely, you will need to run slower than normal production rate; in the North American auto industry we actually run a second evaluation run (we call it “run at rate” to confirm the actual ability to meet the needed pace.
Several suppliers I worked with in China had specialized production teams and areas for running pilot runs. There’s nothing wrong with using specialists to build a small quantity of prototypes, but the actual pilot run should be done by the same workers who will do normal assembly.
Hi Brad,
Agree with all your comments particularly in auto industry and alike. More comprehensive pilot run is definitely the way to to go. Thanks and appreciate your comments.