https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-upgrade-1-28-latest
https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-upgrade-addons-before-controlplane-1-28-latest

keeps failing after kubernetes/release#3254.

/assign

ref #2925

See #2927 (comment) for the conclusion.

• add log and change to use universal deserializer:
  • add diff log to expose the problem: kubeadm: add log for static pod manifest diff kubernetes#120541
  • cleanup kubeadm: fix diff order and add tests for new default value manifest kubernetes#120549 This also changed the unmarshal to use the universal deserializer
• Option1: if we have to revert(import the kubernetes/pkg import is not allowed in kubeadm!)
  • revert Revert #118867: for default unmarshal behavior chang kubernetes#120554 in v1.29 (master)
  • cherry-pick to v1.28 if needed
• Option2: if there is another solution to avoid the defaulting logic
  • kubeadm: remove reference of k8s.io/kubernetes/pkg/apis/core/v1 kubernetes#120561
  • backport to v1.28 as a bug fix. Automated cherry pick of #120561: kubeadm: remove reference of kubernetes#120605
• try to fix the possible problem for users who have upgraded to v1.28.0-2(need to double-check)
  • See kubeadm upgrade from v1.28.0 to v1.28.3 fails #2957 (comment) for more details