Propose Kubeflow to CNCF Incubation Stage

[mchmarny]

Apologies for the shuffle - this PR has been closed and review moved to #1042

---

This PR contains a proposal for Kubeflow project to be accepted into the CNCF Incubation stage.

Kubeflow project is an extendable Machine Learning (ML) platform made of multiple distinct components to address specific stages of the ML lifecycle, including: Mode Development, Model Training, Pipelines, and Multi-framework supports.

Thea Lamkin & Mark Chmarny on behalf of the Kubeflow community

[aronchick]

I just wanted to say how ecstatic I am for this to happen, and how thankful I am about all the folks who worked so hard on getting us to this point. Thank you all so much!

[terrytangyuan]

This is super exciting! Thank you to all the contributors, and looking forward to what we can accomplish next!

[surajkota]

Woot woot!

[jbottum]

Kubeflow’s ambitious goal (e.g. efficient MLOps on Kubernetes) includes a high degree of technical and operational complexity, which requires an on-going dedication to quality, process and open governance. This proposal provides a path for strategic community growth. I appreciate the Community’s (and especially Google’s) commitment to Kubeflow’s open source success with the CNCF.

[cvenets]

We are thrilled with this development! This is going to be a game changer for the Data Science and ML world.

Thank you to Google and everyone in the community who helped get us here.

Onwards and upwards!

[andreeamun]

This is exciting news! It opens new doors to the product and its development is at a turning point. Thank you to everyone who contributed to the project and to Google for its commitment to open-source MLOps.

[thesuperzapper]

This is amazing and a great step forward for the community!

I strongly believe joining an organization like the CNCF will enable Kubeflow to continue growing and maturing as the vendor-neutral home for Kubernetes-based ML tools.

---

@mchmarny I see that we have missed the maintainers of components managed by the Notebooks Working Group and Manifests Working Group in the current proposal.

Components owned by Notebooks Working Group:
(See owners in ./components/OWNERS from the kubeflow/kubeflow repo)

• Admission Webhook (PodDefaults)
• Central Dashboard
• Jupyter Web App
• Kubeflow Access Management API (KFAM)
• Notebook Controller
• Profile Controller
• Tensorboard Controller
• Tensorboard Web App
• Volumes Web App

Components owned by Manifests Working Group:
(See owners in ./OWNERS from the kubeflow/manifests repo)

• Manifests Repository

---

PS: For a full list of working groups that maintain components, see the community page on the Kubeflow website.

[mchmarny]

@hesuperzapper and @jbottum good catch on the missing links. I've applied those in 6092b9c. The roadmap one we can add when that lands in the repo.

[juliusvonkohout]

@jbottum There are plans to create a security workinggroup inside of Kubeflow. Not sure what the status is there and whether this should be added here too.

[akramIOT]

@mchmarny : Big Congratz ! to entire Kubeflow community team for the last 5 years of contributions to take this what started as an Initial Repo based project into CNCF community graduation stage. I have few new requirements/wishlist for the roadmap. thanks a lot.

Questions and New requirements for Kubeflow:

1. Support for deriving Multi and Hybrid Kubeflow data Pipelines with automated annotation of cells from IPYNB and Jupyer Notebooks which is available in Airflow and MLFlow tools as on today.
2. Meta data Tracking which is available in MLFlow today and MLMD, ROK. Tracking run-history and meta data tracking for ~ 10,000 artifacts with Kubeflow. Is this planned in future for Kubeflow ?
3. With HP of Katib and Kale in Kubeflow, can we also plan for automated pipeline caching for 1000’s of runs so that we can archive it in a historical manner for all previous runs ?
4. Manage and browse volumes with a new Volume manager UI, is this planned for future release in Kubeflow ?
5. With Kubeflow’s default Kubeflow’s default tf-operator in use, it leverages OASIS Scheduler in K8’s /Kubernetes for OASIS to calculate the optimal job to be scheduled in a given queue unlike FIFO queue. It does so by calculating required resource and potential utility of each job by deriving a price function. Sometimes this default OASIS scheduling involves additional threshold waiting time for autoscaling and call the correct optimal resource it as needed. This leads to additional overhead of restarting jobs for scaling up and is relatively high with the increase of resource utilization in K8’s cluster. So , Is there any K8’s scheduling related optimization planned from Kubeflow team like identifying Warm, Hot or cold container in different POD’s of K8’s so that dynamic resource scheduling can be done to handle distributed training workloads in K8’s cluster in a more efficient manner ?

[akramIOT]

Does Kubeflow have plan to support any feature like population-based training, which helps improve the likelihood that AI models can reach production readiness quickly for large scale datasets ?

[mchmarny]

Does Kubeflow have plan to support any feature like...

@akramIOT feature requests are most effective when opened as issue in one of the Kubeflow repos, rather than a comment on the PR proposing CNCF incubation.

[jbottum]

@akramIOT - fyi, for population-based training work items, please see, kubeflow/katib#1862

[akramIOT]

Does Kubeflow have plan to support any feature like...

@akramIOT feature requests are most effective when opened as issue in one of the Kubeflow repos, rather than a comment on the PR proposing CNCF incubation.

ok will do a PR @mchmarny , thanks

[rochaporto]

@dims @mchmarny should we schedule a kick-off meeting next week for this? I'll send around a couple proposals.

[dims]

sounds good @rochaporto !

[akgraner]

@dims @rochaporto and @mchmarny Where will the invite to this meeting be shared with the community? I'd like to make sure myself and @jbottum are in attendance.

[tenzen-y]

@mchmarny, Could you update this section like the following since we updated the OWNERS file recently?

https://github.com/kubeflow/katib/blob/f941ec61e5fe8aa17919c51a8b347914ae70b876/OWNERS

    * [Andreyvelich](https://github.com/Andreyvelich) - Andrey Velichkevich (Apple)
    * [Gaocegege](https://github.com/Gaocegege) - Ce Gao (TensorChord)
    * [Johnugeorge](https://github.com/Johnugeorge) - Johnu George (Nutanix)
    * [tenzen-y](https://github.com/tenzen-y) - Yuki Iwai (CyberAgent)

[jbottum]

@mchmarny In the Kubeflow to CNCF transition meeting today, the Community requested a status on this effort per this PR, https://github.com/kubeflow/community/blob/master/proposals/STEERING-COMMITTEE.md.

The Community also identified next steps in today's (Jan 10) meeting which are listed in this doc, https://docs.google.com/document/d/1HXAl6ew5ZUgQaAnEHS1qEPxA5puUz2knUwXOZHU39sA/edit?userstoinvite=mchmarny@google.com&actionButton=1#

How does the Interim Steering Committee propose that we make progress on this project before our next Kubeflow to CNCF Community meeting on Jan 24 ?

[mchmarny]

The CNCF acceptance is gated on TOC review/approval. @dims and @rochaporto any updates on the status of that review?

[joshgav]

I believe you intended to title this section "Model Development" - with an "L" at the end.

[ArangoGutierrez]

+1

[Anarpego]

+1 LET'S FREAKING GOOOOOOO

[rochaporto]

We probably want this reopened? @amye

[jbottum]

@mchmarny questions from KF Community Meeting - can you please re-open this PR ? thanks.

[mrbobbytables]

Replacement PR has been opened here: #1042
Apologies for the shuffle all

[mchmarny]

Thank you, @mrbobbytables.

[amye]

The repository that this came from has been deleted so I can't reopen it, will replace with #1042.