How we made our CICD pipelines 30% faster by putting things in parallel

I wrote this post in July of 2024… which is more than a year ago!

But the lesson still remains true.

Always ask yourself this: What can be done in parallel? What must be done in sequence?

Sometimes low-hanging fruit is really low-hanging. A while ago I managed to reduce our pipelines by 12 minutes (yes 12 minutes) by just putting things in parallel that could be done in parallel.

This was our pipeline before:

I realized that there was no reason whatsoever for both the Package and Copy Model Artifacts step to be in sequence with Build, Test & Analyze, so I put them in sequence!

That’s it. The same pipeline, just 12 minutes and 9 seconds faster.

What can be done in parallel? What must be done in sequence?

Subscribe

I blog about how to grow as a machine learning engineer! Liked this article and want to hear more? Join 40+ others and subscribe!