Skip to content

feat(kaniko): Optimize kaniko build by 50% using compression and add progress#9476

Merged
ericzzzzzzz merged 4 commits into
GoogleContainerTools:mainfrom
idsulik:kaniko-build-optimization
Aug 30, 2024
Merged

feat(kaniko): Optimize kaniko build by 50% using compression and add progress#9476
ericzzzzzzz merged 4 commits into
GoogleContainerTools:mainfrom
idsulik:kaniko-build-optimization

Conversation

@idsulik

@idsulik idsulik commented Jul 18, 2024

Copy link
Copy Markdown
Contributor

Merge current PR after 9464: #9464

Description
We have a large code base(over 96k files) and it's painful to deploy the service because it takes a lot of time. I've spent a lot of time trying to fix this and finally, I did it, now it takes less than 50% time to deploy and it transfers 3x less data.
I also added progress output so it'll be convenient to see what's going on.

User facing changes
New kaniko config option BuildContextCompressionLevel.
New default output Start building with kaniko for artifact.
New progress Sending build context to Kaniko pod {amount}
image.

before optimization:
3 minutes and 40 seconds to create a tar file and 613M transfer
image

after optimization:
1 minute and 24 seconds to create a tar file and 167M transfer
image

@idsulik idsulik force-pushed the kaniko-build-optimization branch 2 times, most recently from 1d1597c to 166adfd Compare July 19, 2024 04:21
@idsulik idsulik requested a review from a team as a code owner July 19, 2024 04:21
@idsulik idsulik requested a review from JeromeJu July 19, 2024 04:21
@idsulik

idsulik commented Jul 19, 2024

Copy link
Copy Markdown
Contributor Author

After both of these changes #9476 and #9451 It now takes only 4 minutes from the time the skaffold dev command is run until the deployment stabilizes.

It used to take more than 15 minutes(~10 minutes for kaniko build and ~5 minutes for helm deploy)

telegram-cloud-photo-size-2-5247051311348833288-x

@idsulik idsulik force-pushed the kaniko-build-optimization branch from 7309250 to c5a33c0 Compare August 28, 2024 04:45
Signed-off-by: Suleiman Dibirov <idsulik@gmail.com>
Signed-off-by: Suleiman Dibirov <idsulik@gmail.com>
Signed-off-by: Suleiman Dibirov <idsulik@gmail.com>
@idsulik idsulik force-pushed the kaniko-build-optimization branch from c5a33c0 to 41cda11 Compare August 29, 2024 15:51
@idsulik

idsulik commented Aug 29, 2024

Copy link
Copy Markdown
Contributor Author

@ericzzzzzzz what do you think about this PR? it reduced kaniko build time by 50% and data transfer by 70%(from 610M to 167M)

@ericzzzzzzz

Copy link
Copy Markdown
Contributor

Hi @idsulik , Thank you so much for the contribution, this is a great improvement. It's gonna take me some time to review this as I'm not very familiar with kaniko.

Signed-off-by: Suleiman Dibirov <idsulik@gmail.com>
@ericzzzzzzz ericzzzzzzz added the kokoro:force-run forces a kokoro re-run on a PR label Aug 30, 2024
@kokoro-team kokoro-team removed the kokoro:force-run forces a kokoro re-run on a PR label Aug 30, 2024
@ericzzzzzzz ericzzzzzzz merged commit 1bc72a7 into GoogleContainerTools:main Aug 30, 2024
@idsulik idsulik deleted the kaniko-build-optimization branch January 14, 2025 12:10
@menahyouyeah menahyouyeah mentioned this pull request Mar 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3 participants