‘ACTION REQUIRED: The Dynamic Resource Allocation (DRA) driver’s DaemonSet
must be deployed with a service account that enables writing ResourceSlice
and reading ResourceClaim objects.’
(kubernetes/kubernetes#125163, @pohly) [SIG Auth, Node and Testing]
Add UserNamespaces field to NodeRuntimeHandlerFeatures (kubernetes/kubernetes#126034, @sohankunkerkar) [SIG API Machinery, Apps and Node]
Added Coordinated Leader Election as Alpha under the CoordinatedLeaderElection feature gate. With the feature enabled, the control plane can use LeaseCandidate objects (coordination.k8s.io/v1alpha1 API group) to participate in a leader election and let the kube-apiserver select the best instance according to some strategy. (kubernetes/kubernetes#124012, @Jefftree) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Release, Scheduling and Testing]
Added a .status.features.supplementalGroupsPolicy field to Nodes. The field is true when the feature is implemented in the CRI implementation (KEP-3619). (kubernetes/kubernetes#125470, @everpeace) [SIG API Machinery, Apps, Node and Testing]
Added an allocatedResourcesStatus to each container status to indicate the health status of devices exposed by the device plugin. (kubernetes/kubernetes#126243, @SergeyKanzhelev) [SIG API Machinery, Apps, Node and Testing]
Added support to the kube-proxy nodePortAddresses / –nodeport-addresses option to
accept the value “primary”, meaning to only listen for NodePort connections
on the node’s primary IPv4 and/or IPv6 address (according to the Node object).
This is strongly recommended, if you were not previously using
–nodeport-addresses, to avoid surprising behavior.
(This behavior is enabled by default with the nftables backend; you would
need to explicitly request --nodeport-addresses 0.0.0.0/0,::/0 there to get
the traditional “listen on all interfaces” behavior.) (kubernetes/kubernetes#123105, @danwinship) [SIG API Machinery, Network and Windows]
Added the feature gates StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks to enforce the strict cost calculation for CEL extended libraries. It is strongly recommended to turn on the feature gates as early as possible. (kubernetes/kubernetes#124675, @cici37) [SIG API Machinery, Auth, Node and Testing]
Changed how the API server handles updates to .spec.defaultBackend of Ingress objects.
Server-side apply now considers .spec.defaultBackend to be an atomic struct. This means that any field-owner who sets values in that struct (they are mutually exclusive) owns the whole struct. For almost all users this change has no impact; for controllers that want to change the default backend port from number to name (or vice-versa), this makes it easier. (kubernetes/kubernetes#126207, @thockin) [SIG API Machinery]
Component-base/logs: when compiled with Go >= 1.21, component-base will automatically configure the slog default logger together with initializing klog. (kubernetes/kubernetes#120696, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing]
CustomResourceDefinition objects created with non-empty caBundle fields which are invalid or do not contain any certificates will not appear in discovery or serve endpoints until a valid caBundle is provided. Updates to CustomResourceDefinition are no longer allowed to transition a valid caBundle field to an invalid caBundle field, because this breaks serving of the existing CustomResourceDefinition. (kubernetes/kubernetes#124061, @Jefftree) [SIG API Machinery]
Dynamic Resource Allocation (DRA): Added a feature so the number of ResourceClaim objects can be limited per namespace and by the number of devices requested through a specific class via the v1.ResourceQuota mechanism. (kubernetes/kubernetes#120611, @pohly) [SIG API Machinery, Apps, Auth, CLI, Etcd, Node, Release, Scheduling and Testing]
Dynamic Resource Allocation (DRA): client-side validation of a ResourceHandle would have accepted a missing DriverName, whereas server-side validation then would have raised an error. (kubernetes/kubernetes#124075, @pohly)
Dynamic Resource Allocation (DRA): in the pod.spec.recourceClaims array, the source indirection is no longer necessary. Instead of e.g. source: resourceClaimTemplateName: my-template, one can write resourceClaimTemplateName: my-template. (kubernetes/kubernetes#125116, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
Enhanced the Dynamic Resource Allocation (DRA) with an updated version of the resource.k8s.io API group. The primary user-facing type remains the ResourceClaim, however significant changes have been made, resulting in the new version, v1alpha3, which is not compatible with the previous version. (kubernetes/kubernetes#125488, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Etcd, Node, Release, Scheduling, Storage and Testing]
Fixed a 1.30.0 regression in OpenAPI descriptions of the imagePullSecrets and
hostAliases fields to mark the fields used as keys in those lists as either defaulted
or required. (kubernetes/kubernetes#124553, @pmalek)
Fixed a 1.30.0 regression in openapi descriptions of PodIP.IP and HostIP.IP fields to mark the fields used as keys in those lists as required. (kubernetes/kubernetes#126057, @thockin)
Fixed a bug in the API server where empty collections of ValidatingAdmissionPolicies did not have an items field. (kubernetes/kubernetes#124568, @xyz-li) [SIG API Machinery]
Fixed a deep copy issue when retrieving the controller reference. (kubernetes/kubernetes#124116, @HiranmoyChowdhury) [SIG API Machinery and Release]
Fixed code-generator client-gen to work with api/v1-like package structure. (kubernetes/kubernetes#125162, @sttts) [SIG API Machinery and Apps]
Fixed incorrect “v1 Binding is deprecated in v1.6+” warning in kube-scheduler log. (kubernetes/kubernetes#125540, @pohly) [SIG API Machinery]
Fixed the comment for the Job’s managedBy field. (kubernetes/kubernetes#124793, @mimowo) [SIG API Machinery and Apps]
Fixed the documentation for the default value of the procMount entry in securityContext within a Pod.
The documentation was previously using the name of the internal variable DefaultProcMount, rather than the actual value, “Default”. (kubernetes/kubernetes#125782, @aborrero) [SIG Apps and Node]
Graduate PodDisruptionConditions to GA and lock (kubernetes/kubernetes#125461, @mimowo) [SIG Apps, Node, Scheduling and Testing]
Graduated MatchLabelKeys/MismatchLabelKeys feature in PodAffinity/PodAntiAffinity to Beta. (kubernetes/kubernetes#123638, @sanposhiho) [SIG API Machinery, Apps, Scheduling and Testing]
Graduated JobPodFailurePolicy to GA and locked it to it’s default. (kubernetes/kubernetes#125442, @mimowo) [SIG API Machinery, Apps, Scheduling and Testing]
Graduated the Job successPolicy field to beta.
The new reason label, “SuccessPolicy” and “CompletionsReached” are added to the “jobs_finished_total” metric.
Additionally, if you enable the JobSuccessPolicy feature gate, the Job gets “CompletionsReached” reason for the “SuccessCriteriaMet” and “Complete” condition type
when the number of succeeded Job Pods (.status.succeeded) reached the desired completions (.spec.completions). (kubernetes/kubernetes#126067, @tenzen-y) [SIG API Machinery, Apps and Testing]
Graduated the DisableNodeKubeProxyVersion feature gate to beta. By default, the kubelet no longer attempts to set the .status.kubeProxyVersion field for its associated Node. (kubernetes/kubernetes#123845, @HirazawaUi) [SIG API Machinery, Cloud Provider, Network, Node and Testing]
Improved scheduling performance when many nodes, and prefilter returned 1-2 nodes (e.g. daemonset)
For developers of out-of-tree PostFilter plugins, note that the semantics of NodeToStatusMap are changing: A node with an absent value in the NodeToStatusMap should be interpreted as having an UnschedulableAndUnresolvable status. (kubernetes/kubernetes#125197, @gabesaba)
Introduced a new boolean kubelet flag --fail-cgroupv1. (kubernetes/kubernetes#126031, @harche) [SIG API Machinery and Node]
K8s.io/apimachinery/pkg/util/runtime: Added support for new calls to handle panics and errors in the context where they occur. PanicHandlers and ErrorHandlers now must accept a context parameter for that. Log output is structured instead of unstructured. (kubernetes/kubernetes#121970, @pohly) [SIG API Machinery and Instrumentation]
KEP-1880: Users of the new feature to add multiple service CIDR will use by default a dual-write strategy on the new ClusterIP allocators to avoid the problem of possible duplicate IPs allocated to Services when running skewed kube-apiservers using different allocators. They can opt-out of this behavior by enabled the feature gate DisableAllocatorDualWrite. (kubernetes/kubernetes#122047, @aojea) [SIG API Machinery, Apps, Instrumentation and Testing]
Kube-apiserver: Added Alpha features to allow API server authz to check the context of requests:
- The
AuthorizeWithSelectors feature gate enables including field and label selector information from requests in webhook authorization calls.
- The
AuthorizeNodeWithSelectors feature gate changes node authorizer behavior to limit requests from node API clients, so that each Node can only get / list / watch its own Node API object, and can also only get / list / watch Pod API objects bound to that node. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or obtain broader read access independent of the node authorizer. (kubernetes/kubernetes#125571, @liggitt) [SIG API Machinery, Auth, Node, Scheduling and Testing]
Kube-apiserver: ControllerRevision objects are now verified to contain valid JSON data in the data field. (kubernetes/kubernetes#125549, @liggitt) [SIG API Machinery and Apps]
Kube-apiserver: the --encryption-provider-config file is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. When --encryption-provider-config-automatic-reload is used, new encryption config files that contain typos after the kube-apiserver is running are treated as invalid and the last valid config is used. (kubernetes/kubernetes#124912, @enj) [SIG API Machinery and Auth]
Kube-controller-manager: the horizontal-pod-autoscaler-upscale-delay and horizontal-pod-autoscaler-downscale-delay flags have been removed (deprecated and non-functional since v1.12). (kubernetes/kubernetes#124948, @SataQiu) [SIG API Machinery, Apps and Autoscaling]
Made kube-proxy Windows service control manager integration (--windows-service) configurable in v1alpha1 component configuration via windowsRunAsService field. (kubernetes/kubernetes#126072, @aroradaman) [SIG Network and Scalability]
PersistentVolumeLastPhaseTransitionTime feature is stable and enabled by default. (kubernetes/kubernetes#124969, @RomanBednar) [SIG API Machinery, Apps, Storage and Testing]
Promoted LocalStorageCapacityIsolation to beta; the behaviour is enabled by default. Within the kubelet, storage capacity isolation is active if the feature gate is enabled and the specific Pod is using a user namespace. (kubernetes/kubernetes#126014, @PannagaRao) [SIG Apps, Autoscaling, Node, Storage and Testing]
Promoted StatefulSetStartOrdinal to stable. This means --feature-gates=StatefulSetStartOrdinal=true are not needed on kube-apiserver and kube-controller-manager binaries and they’ll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation. (kubernetes/kubernetes#125374, @pwschuurman) [SIG API Machinery, Apps and Testing]
Promoted feature-gate VolumeAttributesClass to beta (disabled by default). Users need to enable the feature gate and the storage.k8s.io/v1beta1 API group to use this feature.
Promoted the VolumeAttributesClass API to beta. (kubernetes/kubernetes#126145, @carlory) [SIG API Machinery, Apps, CLI, Etcd, Storage and Testing]
Removed deprecated command flags –volume-host-cidr-denylist
and –volume-host-allow-local-loopback from kube-controller-manager.
(kubernetes/kubernetes#124017, @carlory) [SIG API Machinery, Apps, Cloud Provider and Storage]
Removed feature gate CustomResourceValidationExpressions. (kubernetes/kubernetes#126136, @cici37) [SIG API Machinery, Cloud Provider and Testing]
Reverted a change where ConsistentListFromCache was moved to beta and enabled by default. (kubernetes/kubernetes#126139, @enj)
Revised the Pod API with Alpha support for volumes derived from OCI artifacts. This feature is behind the ImageVolume feature gate. (kubernetes/kubernetes#125660, @saschagrunert) [SIG API Machinery, Apps and Node]
Supported fine-grained supplemental groups policy (KEP-3619), which enabled
fine-grained control for supplementary groups in the first container processes.
This allows you to choose whether to include groups defined in the container image (/etc/groups)
for the container’s primary UID or not. (kubernetes/kubernetes#117842, @everpeace) [SIG API Machinery, Apps and Node]
The (alpha) nftables mode of kube-proxy now requires version 1.0.1 or later
of the nft command-line, and kernel 5.13 or later. (For testing/development
purposes, you can use older kernels, as far back as 5.4, if you set the
nftables.skipKernelVersionCheck option in the kube-proxy config, but this is not
recommended in production since it may cause problems with other nftables
users on the system.) (kubernetes/kubernetes#124152, @danwinship) [SIG Network]
To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions for existing expressions in storage, will fully support runtime in next release for compatibility concern. (kubernetes/kubernetes#126188, @cici37) [SIG API Machinery and Testing]
Updated the feature MultiCIDRServiceAllocator to beta (disabled by default). Users need to enable the feature gate and the networking v1beta1 group to be able to use this new feature, that allows to dynamically reconfigure Service CIDR ranges. (kubernetes/kubernetes#125021, @aojea) [SIG API Machinery, Apps, CLI, Etcd, Instrumentation, Network and Testing]
Use omitempty for optional Job Pod Failure Policy fields. (kubernetes/kubernetes#126046, @mimowo)
User can choose a different static policy option SpreadPhysicalCPUsPreferredOption to spread cpus across physical cpus for some specific applications (kubernetes/kubernetes#123733, @Jeffwan) [SIG Node]
When the featuregate AnonymousAuthConfigurableEndpoints is enabled users can update the AuthenticationConfig file with endpoints for with anonymous requests are alllowed. (kubernetes/kubernetes#124917, @vinayakankugoyal) [SIG API Machinery, Auth, Cloud Provider, Node and Testing]
Move ConsistentListFromCache feature flag to Beta and enable it by default (kubernetes/kubernetes#126469, @serathius) [SIG API Machinery]
Add Coordinated Leader Election as alpha under the CoordinatedLeaderElection feature gate. With the feature enabled, the control plane can use LeaseCandidate objects (coordination.k8s.io/v1alpha1 API group) to participate in a leader election and let the kube-apiserver select the best instance according to some strategy. (kubernetes/kubernetes#124012, @Jefftree) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Release, Scheduling and Testing]
Add an AllocatedResourcesStatus to each container status to indicate the health status of devices exposed by the device plugin. (kubernetes/kubernetes#126243, @SergeyKanzhelev) [SIG API Machinery, Apps, Node and Testing]
Added Node.Status.Features.SupplementalGroupsPolicy field which is set to true when the feature is implemented in the CRI implementation (KEP-3619) (kubernetes/kubernetes#125470, @everpeace) [SIG API Machinery, Apps, Node and Testing]
CustomResourceDefinition objects created with non-empty caBundle fields which are invalid or do not contain any certificates will not appear in discovery or serve endpoints until a valid caBundle is provided. Updates to CustomResourceDefinition are no longer allowed to transition a valid caBundle field to an invalid caBundle field. (kubernetes/kubernetes#124061, @Jefftree) [SIG API Machinery]
DRA: The DRA driver’s daemonset must be deployed with a service account that enables writing ResourceSlice and reading ResourceClaim objects. (kubernetes/kubernetes#125163, @pohly) [SIG Auth, Node and Testing]
DRA: new API and several new features (kubernetes/kubernetes#125488, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Etcd, Node, Release, Scheduling, Storage and Testing]
DRA: the number of ResourceClaim objects can be limited per namespace and by the number of devices requested through a specific class via the v1.ResourceQuota mechanism. (kubernetes/kubernetes#120611, @pohly) [SIG API Machinery, Apps, Auth, CLI, Etcd, Node, Release, Scheduling and Testing]
Fix the documentation for the default value of the procMount entry in the pod securityContext.
The documentation was previously using the name of the internal variable ‘DefaultProcMount’ rather than the actual value ‘Default’. (kubernetes/kubernetes#125782, @aborrero) [SIG Apps and Node]
Fixed a bug in the API server where empty collections of ValidatingAdmissionPolicies did not have an items field. (kubernetes/kubernetes#124568, @xyz-li) [SIG API Machinery]
Graduate the Job SuccessPolicy to Beta.
The new reason label, “SuccessPolicy” and “CompletionsReached” are added to the “jobs_finished_total” metric.
Additionally, If we enable the “JobSuccessPolicy” feature gate, the Job gets “CompletionsReached” reason for the “SuccessCriteriaMet” and “Complete” condition type
when the number of succeeded Job Pods (“.status.succeeded”) reached the desired completions (“.spec.completions”). (kubernetes/kubernetes#126067, @tenzen-y) [SIG API Machinery, Apps and Testing]
Introduce a new boolean kubelet flag –fail-cgroupv1 (kubernetes/kubernetes#126031, @harche) [SIG API Machinery and Node]
Kube-apiserver: adds an alpha AuthorizeWithSelectors feature that includes field and label selector information from requests in webhook authorization calls; adds an alpha AuthorizeNodeWithSelectors feature that makes the node authorizer limit requests from node API clients to get / list / watch its own Node API object, and to get / list / watch its own Pod API objects. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or grant broader read access independent of the node authorizer. (kubernetes/kubernetes#125571, @liggitt) [SIG API Machinery, Auth, Node, Scheduling and Testing]
Kube-proxy Windows service control manager integration(–windows-service) is now configurable in v1alpha1 component configuration via WindowsRunAsService field (kubernetes/kubernetes#126072, @aroradaman) [SIG Network and Scalability]
Promote LocalStorageCapacityIsolation to beta and enable if user namespace is enabled for the pod (kubernetes/kubernetes#126014, @PannagaRao) [SIG Apps, Autoscaling, Node, Storage and Testing]
Promote StatefulSetStartOrdinal to stable. This means --feature-gates=StatefulSetStartOrdinal=true are not needed on kube-apiserver and kube-controller-manager binaries and they’ll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation (kubernetes/kubernetes#125374, @pwschuurman) [SIG API Machinery, Apps and Testing]
Promoted feature-gate VolumeAttributesClass to beta (disabled by default). Users need to enable the feature gate and the storage v1beta1 group to use this new feature.
- Promoted API
VolumeAttributesClass and VolumeAttributesClassList to storage.k8s.io/v1beta1. (kubernetes/kubernetes#126145, @carlory) [SIG API Machinery, Apps, CLI, Etcd, Storage and Testing]
Removed feature gate CustomResourceValidationExpressions. (kubernetes/kubernetes#126136, @cici37) [SIG API Machinery, Cloud Provider and Testing]
Revert “Move ConsistentListFromCache feature flag to Beta and enable it by default” (kubernetes/kubernetes#126139, @enj) [SIG API Machinery]
Revised the Pod API with alpha support for volumes derived from OCI artefacts.
This feature is behind the ImageVolume feature gate. (kubernetes/kubernetes#125660, @saschagrunert) [SIG API Machinery, Apps and Node]
The Ingress.spec.defaultBackend is now considered an atomic struct for the purposes of server-side-apply. This means that any field-owner who sets values in that struct (they are mutually exclusive) owns the whole struct. For almost all users this change has no impact. For controllers which want to change port from number to name (or vice-versa), this makes it easier. (kubernetes/kubernetes#126207, @thockin) [SIG API Machinery]
To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions for existing expressions in storage, will fully support runtime in next release for compatibility concern. (kubernetes/kubernetes#126188, @cici37) [SIG API Machinery and Testing]
Add UserNamespaces field to NodeRuntimeHandlerFeatures (kubernetes/kubernetes#126034, @sohankunkerkar) [SIG API Machinery, Apps and Node]
Fixes a 1.30.0 regression in openapi descriptions of PodIP.IP and HostIP.IP fields to mark the fields used as keys in those lists as required. (kubernetes/kubernetes#126057, @thockin) [SIG API Machinery]
Graduate JobPodFailurePolicy to GA and lock (kubernetes/kubernetes#125442, @mimowo) [SIG API Machinery, Apps, Scheduling and Testing]
Graduate PodDisruptionConditions to GA and lock (kubernetes/kubernetes#125461, @mimowo) [SIG Apps, Node, Scheduling and Testing]
PersistentVolumeLastPhaseTransitionTime feature is stable and enabled by default. (kubernetes/kubernetes#124969, @RomanBednar) [SIG API Machinery, Apps, Storage and Testing]
The (alpha) nftables mode of kube-proxy now requires version 1.0.1 or later
of the nft command-line, and kernel 5.13 or later. (For testing/development
purposes, you can use older kernels, as far back as 5.4, if you set the
nftables.skipKernelVersionCheck option in the kube-proxy config, but this is not
recommended in production since it may cause problems with other nftables
users on the system.) (kubernetes/kubernetes#124152, @danwinship) [SIG Network]
Use omitempty for optional Job Pod Failure Policy fields (kubernetes/kubernetes#126046, @mimowo) [SIG Apps]
User can choose a different static policy option SpreadPhysicalCPUsPreferredOption to spread cpus across physical cpus for some specific applications (kubernetes/kubernetes#123733, @Jeffwan) [SIG Node]
DRA: in the pod.spec.recourceClaims array, the source indirection is no longer necessary. Instead of e.g. source: resourceClaimTemplateName: my-template, one can write resourceClaimTemplateName: my-template. (kubernetes/kubernetes#125116, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
Fix code-generator client-gen to work with api/v1-like package structure. (kubernetes/kubernetes#125162, @sttts) [SIG API Machinery and Apps]
KEP-1880: Users of the new feature to add multiple service CIDR will use by default a dual-write strategy on the new ClusterIP allocators to avoid the problem of possible duplicate IPs allocated to Services when running skewed kube-apiservers using different allocators. They can opt-out of this behavior by enabled the feature gate DisableAllocatorDualWrite (kubernetes/kubernetes#122047, @aojea) [SIG API Machinery, Apps, Instrumentation and Testing]
Kube-apiserver: ControllerRevision objects are now verified to contain valid JSON data in the data field. (kubernetes/kubernetes#125549, @liggitt) [SIG API Machinery and Apps]
Update the feature MultiCIDRServiceAllocator to beta (disabled by default). Users need to enable the feature gate and the networking v1beta1 group to be able to use this new feature, that allows to dynamically reconfigure Service CIDR ranges. (kubernetes/kubernetes#125021, @aojea) [SIG API Machinery, Apps, CLI, Etcd, Instrumentation, Network and Testing]
When the featuregate AnonymousAuthConfigurableEndpoints is enabled users can update the AuthenticationConfig file with endpoints for with anonymous requests are alllowed. (kubernetes/kubernetes#124917, @vinayakankugoyal) [SIG API Machinery, Auth, Cloud Provider, Node and Testing]
Fixed incorrect “v1 Binding is deprecated in v1.6+” warning in kube-scheduler log. (kubernetes/kubernetes#125540, @pohly) [SIG API Machinery]
Added the feature gates StrictCostEnforcementForVAP and StrictCostEnforcementForWebhooks to enforce the strct cost calculation for CEL extended libraries. It is strongly recommended to turn on the feature gates as early as possible. (kubernetes/kubernetes#124675, @cici37) [SIG API Machinery, Auth, Node and Testing]
Component-base/logs: when compiled with Go >= 1.21, component-base will automatically configure the slog default logger together with initializing klog. (kubernetes/kubernetes#120696, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing]
DRA: client-side validation of a ResourceHandle would have accepted a missing DriverName, whereas server-side validation then would have raised an error. (kubernetes/kubernetes#124075, @pohly) [SIG Apps]
Fix Deep Copy issue in getting controller reference (kubernetes/kubernetes#124116, @HiranmoyChowdhury) [SIG API Machinery and Release]
Fix the comment for the Job’s managedBy field (kubernetes/kubernetes#124793, @mimowo) [SIG API Machinery and Apps]
Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. (kubernetes/kubernetes#124553, @pmalek) [SIG API Machinery]
Graduate MatchLabelKeys/MismatchLabelKeys feature in PodAffinity/PodAntiAffinity to Beta (kubernetes/kubernetes#123638, @sanposhiho) [SIG API Machinery, Apps, Scheduling and Testing]
Graduated the DisableNodeKubeProxyVersion feature gate to beta. By default, the kubelet no longer attempts to set the .status.kubeProxyVersion field for its associated Node. (kubernetes/kubernetes#123845, @HirazawaUi) [SIG API Machinery, Cloud Provider, Network, Node and Testing]
Improved scheduling performance when many nodes, and prefilter returns 1-2 nodes (e.g. daemonset)
For developers of out-of-tree PostFilter plugins, note that the semantics of NodeToStatusMap are changing: A node with an absent value in the NodeToStatusMap should be interpreted as having an UnschedulableAndUnresolvable status (kubernetes/kubernetes#125197, @gabesaba) [SIG Scheduling]
K8s.io/apimachinery/pkg/util/runtime: new calls support handling panics and errors in the context where they occur. PanicHandlers and ErrorHandlers now must accept a context parameter for that. Log output is structured instead of unstructured. (kubernetes/kubernetes#121970, @pohly) [SIG API Machinery and Instrumentation]
Kube-apiserver: the --encryption-provider-config file is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. When --encryption-provider-config-automatic-reload is used, new encryption config files that contain typos after the kube-apiserver is running are treated as invalid and the last valid config is used. (kubernetes/kubernetes#124912, @enj) [SIG API Machinery and Auth]
Kube-controller-manager removes deprecated command flags: –volume-host-cidr-denylist and –volume-host-allow-local-loopback (kubernetes/kubernetes#124017, @carlory) [SIG API Machinery, Apps, Cloud Provider and Storage]
Kube-controller-manager: the horizontal-pod-autoscaler-upscale-delay and horizontal-pod-autoscaler-downscale-delay flags have been removed (deprecated and non-functional since v1.12) (kubernetes/kubernetes#124948, @SataQiu) [SIG API Machinery, Apps and Autoscaling]
Support fine-grained supplemental groups policy (KEP-3619), which enables fine-grained control for supplementary groups in the first container processes. You can choose whether to include groups defined in the container image(/etc/groups) for the container’s primary uid or not. (kubernetes/kubernetes#117842, @everpeace) [SIG API Machinery, Apps and Node]
The kube-proxy nodeportAddresses / –nodeport-addresses option now
accepts the value “primary”, meaning to only listen for NodePort connections
on the node’s primary IPv4 and/or IPv6 address (according to the Node object).
This is strongly recommended, if you were not previously using
–nodeport-addresses, to avoid surprising behavior.
(This behavior is enabled by default with the nftables backend; you would
need to explicitly request --nodeport-addresses 0.0.0.0/0,::/0 there to get
the traditional “listen on all interfaces” behavior.) (kubernetes/kubernetes#123105, @danwinship) [SIG API Machinery, Network and Windows]
Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. (kubernetes/kubernetes#124553, @pmalek) [SIG API Machinery]
Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. (kubernetes/kubernetes#124694, @pmalek) [SIG API Machinery]
Added (alpha) support for the managedBy field on Jobs. Jobs with a custom value of this field - any value other than kubernetes.io/job-controller - were skipped by the job controller, and their reconciliation was delegated to an external controller, indicated by the value of the field. Jobs that didn’t have this field at all, or where the field value was the reserved string kubernetes.io/job-controller, were reconciled by the built-in job controller.
(kubernetes/kubernetes#123273, @mimowo)
Added alpha-level support for the SuccessPolicy in Jobs.
(kubernetes/kubernetes#123412, @tenzen-y)
Added the CEL library for IP Addresses and CIDRs. This was made available for use starting from version 1.31.
(kubernetes/kubernetes#121912, @JoelSpeed)
Allowed container runtimes to fix an image garbage collection bug by adding an image_id field to the CRI Container message.
(kubernetes/kubernetes#123508, @saschagrunert)
Dynamic Resource Allocation: DRA drivers can now use “structured parameters” to let the scheduler handle claim allocation.
(kubernetes/kubernetes#123516, @pohly)
Fixed accidental enablement of the new alpha optionalOldSelf API field in CustomResourceDefinition validation rules, which should only have been allowed to be set when the CRDValidationRatcheting feature gate is enabled.
(kubernetes/kubernetes#122329, @jpbetz)
Implemented the prescore extension point for the volumeBinding plugin. It now returns skip if it doesn’t do anything in Score.
(kubernetes/kubernetes#115768, @AxeZhan)
Kubelet would fail if NodeSwap was used with LimitedSwap and cgroupv1 node.
(kubernetes/kubernetes#123738, @kannon92)
Promoted AdmissionWebhookMatchConditions to GA. The feature is now stable, and the feature gate is now locked to default.
(kubernetes/kubernetes#123560, @ivelichkovich)
Structured Authentication Configuration now supports DiscoveryURL. If specified, discoveryURL overrides the URL used to fetch discovery information. This is for scenarios where the well-known and jwks endpoints are hosted at a different location than the issuer (such as locally in the cluster).
(kubernetes/kubernetes#123527, @aramase)
The StorageVersionMigration API, previously available as a Custom Resource Definition (CRD), is now a built-in API in Kubernetes.
(kubernetes/kubernetes#123344, @nilekhc)
When configuring a JWT authenticator:
If username.expression used ‘claims.email’, then ‘claims.email_verified’ must have been used in username.expression or extra[*].valueExpression or claimValidationRules[*].expression. An example claim validation rule expression that matches the validation automatically applied when username.claim is set to ‘email’ is ‘claims.?email_verified.orValue(true)’.
(kubernetes/kubernetes#123737, @enj)
readOnly volumes now support recursive read-only mounts for kernel versions >= 5.12.”
(kubernetes/kubernetes#123180, @AkihiroSuda)
cri-api: Implemented KEP-3857: Recursive Read-only (RRO) mounts.
(kubernetes/kubernetes#123272, @AkihiroSuda)
kube-apiserver: the AuthenticationConfiguration type accepted in --authentication-config files has been promoted to apiserver.config.k8s.io/v1beta1.
(kubernetes/kubernetes#123696, @aramase)
kubelet allowed specifying a custom root directory for pod logs (instead of the default /var/log/pods) using the podLogsDir key in kubelet configuration.
(kubernetes/kubernetes#112957, @mxpv)
resource.k8s.io/ResourceClaim (alpha API): The strategic merge patch strategy for the status.reservedFor array was changed so that a strategic-merge-patch can now add individual entries. This change may break clients using strategic merge patch to update status, which rely on the previous behavior (replacing the entire array).
(kubernetes/kubernetes#122276, @pohly)
Added a CBOR implementation of runtime.Serializer. Until CBOR graduates to Alpha, API servers will refuse to start if configured with CBOR support. (kubernetes/kubernetes#122881, @benluddy)
Added a alpha feature, behind the RelaxedEnvironmentVariableValidation feature gate.
When that gate is enabled, Kubernetes allows almost all printable ASCII characters to be used in the names
of environment variables for containers in Pods. (kubernetes/kubernetes#123385, @HirazawaUi)
Added a new (alpha) field, trafficDistribution, to the Service spec to express preferences for traffic distribution to endpoints. Enabled through the ServiceTrafficDistribution feature gate. (kubernetes/kubernetes#123487, @gauravkghildiyal)
Added audienceMatchPolicy field to AuthenticationConfiguration and support for configuring multiple audiences.
The “audienceMatchPolicy” can be empty (or unset) when a single audience is specified in the “audiences” field.
The “audienceMatchPolicy” must be set to “MatchAny” when multiple audiences are specified in the “audiences” field. (kubernetes/kubernetes#123165, @aramase)
Added consistent vanity import to files and provided tooling for verifying and updating them. (kubernetes/kubernetes#120642, @jcchavezs)
Added the disable-force-detach CLI option for kube-controller-manager. By default, it’s set to false. When enabled, it prevents force detaching volumes based on maximum unmount time and node status. If activated, the non-graceful node shutdown feature must be used to recover from node failure. Additionally, if a pod needs to be forcibly terminated at the risk of corruption, the appropriate VolumeAttachment object must be deleted. (kubernetes/kubernetes#120344, @rohitssingh)
Added to MutableFeatureGate the ability to override the default setting of feature gates, to allow default-enabling a feature on a component-by-component basis instead of for all affected components simultaneously. (kubernetes/kubernetes#122647, @benluddy)
Aggregated discovery supports both v2beta1 and v2 types and feature is promoted to GA. (kubernetes/kubernetes#122882, @Jefftree)
Alpha support for field selectors on custom resources has been added. With the CustomResourceFieldSelectors feature gate enabled, the CustomResourceDefinition API now allows specifying selectableFields. Listing a field there enables filtering custom resources for that CustomResourceDefinition in list or watch requests. (kubernetes/kubernetes#122717, @jpbetz)
AppArmor profiles can now be configured through fields on the PodSecurityContext and container SecurityContext. The beta AppArmor annotations are deprecated, and AppArmor status is no longer included in the node ready condition. (kubernetes/kubernetes#123435, @tallclair)
Contextual logging is now in beta and enabled by default. Check out the KEP and official documentation for more details. (kubernetes/kubernetes#122589, @pohly)
Enabled concurrent log rotation in kubelet. You can now configure the maximum number of concurrent rotations with the containerLogMaxWorkers setting, and adjust the monitoring interval with containerLogMonitorInterval. (kubernetes/kubernetes#114301, @harshanarayana)
Graduated pod scheduling gates to general availability.
The PodSchedulingReadiness feature gate no longer has any effect, and the
.spec.schedulingGates field is always available within the Pod and PodTemplate APIs. (kubernetes/kubernetes#123575, @Huang-Wei)
Graduated support for minDomains in pod topology spread constraints, to general availability.
The MinDomainsInPodTopologySpread feature gate no longer has any effect, and the field is
always available within the Pod and PodTemplate APIs. (kubernetes/kubernetes#123481, @sanposhiho)
In kubelet configuration, the .memorySwap.swapBehavior field now accepts a new value NoSwap, which becomes the default if unspecified. The previously accepted UnlimitedSwap value has been dropped.
(kubernetes/kubernetes#122745, @kannon92)
Kube-apiserver: the AuthorizationConfiguration type accepted in --authorization-config files has been promoted to apiserver.config.k8s.io/v1beta1. (kubernetes/kubernetes#123640, @liggitt)
OIDC authentication will now fail if the username asserted based on a CEL expression config is the empty string. Previously the request would be authenticated with the username set to the empty string. (kubernetes/kubernetes#123568, @enj)
Removed note that hostAliases are not supported on hostNetwork Pods from the PodSpec API. The feature has been supported since v1.8. (kubernetes/kubernetes#122422, @neolit123)
Structured Authentication Configuration now supports configuring multiple JWT authenticators. The maximum allowed JWT authenticators in the authentication configuration is 64. (kubernetes/kubernetes#123431, @aramase)
Text logging in Kubernetes components now uses textlogger. The same split streams of info and error log entries with buffering of info entries is now also supported for text output (off by default, alpha feature). Previously, this was only supported for JSON. Performance is better also without split streams. (kubernetes/kubernetes#114672, @pohly)
The API server now detects and fails on startup if there are conflicting issuers between JWT authenticators and service account configurations. Previously, such configurations would run but could be inconsistently effective depending on the credential. (kubernetes/kubernetes#123561, @enj)
The JWT authenticator configuration set via the --authentication-config flag is now dynamically reloaded as the file changes on disk. (kubernetes/kubernetes#123525, @enj)
The StructuredAuthenticationConfiguration feature is now beta and enabled. (kubernetes/kubernetes#123719, @enj)
The kube_codegen tool now ignores the vendor folder during code generation.
(kubernetes/kubernetes#122729, @jparrill)
The kubernetes repo now uses Go workspaces. This should not impact end users at all, but does have impact for developers of downstream projects. Switching to workspaces caused some breaking changes in the flags to the various k8s.io/code-generator tools. Downstream consumers should look at staging/src/k8s.io/code-generator/kube_codegen.sh to see the changes. (kubernetes/kubernetes#123529, @thockin)
Updated an audit annotation key used by the …/serviceaccounts/<name>/token resource handler.
The annotation used to persist the issued credential identifier is now authentication.kubernetes.io/issued-credential-id. (kubernetes/kubernetes#123098, @munnerz) [SIG Auth]
Users are now allowed to mutate FSGroupPolicy and PodInfoOnMount in CSIDriver.Spec. (kubernetes/kubernetes#116209, @haoruan)
ValidatingAdmissionPolicy was promoted to GA and will be enabled by default. (kubernetes/kubernetes#123405, @cici37)
When scheduling a mix of pods using ResourceClaims and others that don’t, scheduling a pod with ResourceClaims has a lower impact on scheduling latency. (kubernetes/kubernetes#121876, @pohly)
When working with client-go events, it’s now recommended to use NewEventBroadcasterAdapterWithContext instead of NewEventBroadcasterAdapter if contextual logging support is needed. (kubernetes/kubernetes#122142, @pohly)
A new (alpha) field, trafficDistribution, has been added to the Service spec.
This field provides a way to express preferences for how traffic is distributed to the endpoints for a Service.
It can be enabled through the ServiceTrafficDistribution feature gate. (kubernetes/kubernetes#123487, @gauravkghildiyal) [SIG API Machinery, Apps and Network]
Add alpha-level support for the SuccessPolicy in Jobs (kubernetes/kubernetes#123412, @tenzen-y) [SIG API Machinery, Apps and Testing]
Added (alpha) support for the managedBy field on Jobs. Jobs with a custom value of this field - any
value other than kubernetes.io/job-controller - are skipped by the job controller, and their
reconciliation is delegated to an external controller, indicated by the value of the field. Jobs that
don’t have this field at all, or where the field value is the reserved string kubernetes.io/job-controller,
are reconciled by the built-in job controller. (kubernetes/kubernetes#123273, @mimowo) [SIG API Machinery, Apps and Testing]
Added a alpha feature, behind the RelaxedEnvironmentVariableValidation feature gate.
When that gate is enabled, Kubernetes allows almost all printable ASCII characters to be used in the names
of environment variables for containers in Pods. (kubernetes/kubernetes#123385, @HirazawaUi) [SIG Apps, Node and Testing]
Added alpha support for field selectors on custom resources.
Provided that the CustomResourceFieldSelectors feature gate is enabled, the CustomResourceDefinition
API now lets you specify selectableFields. Listing a field there allows filtering custom resources for that
CustomResourceDefinition in list or watch requests. (kubernetes/kubernetes#122717, @jpbetz) [SIG API Machinery]
Added support for configuring multiple JWT authenticators in Structured Authentication Configuration. The maximum allowed JWT authenticators in the authentication configuration is 64. (kubernetes/kubernetes#123431, @aramase) [SIG Auth and Testing]
Aggregated discovery supports both v2beta1 and v2 types and feature is promoted to GA (kubernetes/kubernetes#122882, @Jefftree) [SIG API Machinery and Testing]
Allowing container runtimes to fix an image garbage collection bug by adding an image_id field to the CRI Container message. (kubernetes/kubernetes#123508, @saschagrunert) [SIG Node]
AppArmor profiles can now be configured through fields on the PodSecurityContext and container SecurityContext.
- The beta AppArmor annotations are deprecated.
- AppArmor status is no longer included in the node ready condition (kubernetes/kubernetes#123435, @tallclair) [SIG API Machinery, Apps, Auth, Node and Testing]
Conflicting issuers between JWT authenticators and service account config are now detected and fail on API server startup. Previously such a config would run but would be inconsistently effective depending on the credential. (kubernetes/kubernetes#123561, @enj) [SIG API Machinery and Auth]
Dynamic Resource Allocation: DRA drivers may now use “structured parameters” to let the scheduler handle claim allocation. (kubernetes/kubernetes#123516, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Instrumentation, Node, Release, Scheduling, Storage and Testing]
Graduated pod scheduling gates to general availability.
The PodSchedulingReadiness feature gate no longer has any effect, and the
.spec.schedulingGates field is always available within the Pod and PodTemplate APIs. (kubernetes/kubernetes#123575, @Huang-Wei) [SIG API Machinery, Apps, Node, Scheduling and Testing]
Graduated support for minDomains in pod topology spread constraints, to general availability.
The MinDomainsInPodTopologySpread feature gate no longer has any effect, and the field is
always available within the Pod and PodTemplate APIs. (kubernetes/kubernetes#123481, @sanposhiho) [SIG API Machinery, Apps, Scheduling and Testing]
JWT authenticator config set via the –authentication-config flag is now dynamically reloaded as the file changes on disk. (kubernetes/kubernetes#123525, @enj) [SIG API Machinery, Auth and Testing]
Kube-apiserver: the AuthenticationConfiguration type accepted in --authentication-config files has been promoted to apiserver.config.k8s.io/v1beta1. (kubernetes/kubernetes#123696, @aramase) [SIG API Machinery, Auth and Testing]
Kube-apiserver: the AuthorizationConfiguration type accepted in --authorization-config files has been promoted to apiserver.config.k8s.io/v1beta1. (kubernetes/kubernetes#123640, @liggitt) [SIG Auth and Testing]
Kubelet should fail if NodeSwap is used with LimitedSwap and cgroupv1 node. (kubernetes/kubernetes#123738, @kannon92) [SIG API Machinery, Node and Testing]
Kubelet: a custom root directory for pod logs (instead of default /var/log/pods) can be specified using the podLogsDir
key in kubelet configuration. (kubernetes/kubernetes#112957, @mxpv) [SIG API Machinery, Node, Scalability and Testing]
Kubelet: the .memorySwap.swapBehavior field in kubelet configuration accepts a new value NoSwap and makes this the default if unspecified; the previously accepted UnlimitedSwap value has been dropped. (kubernetes/kubernetes#122745, @kannon92) [SIG API Machinery, Node and Testing]
OIDC authentication will now fail if the username asserted based on a CEL expression config is the empty string. Previously the request would be authenticated with the username set to the empty string. (kubernetes/kubernetes#123568, @enj) [SIG API Machinery, Auth and Testing]
PodSpec API: remove note that hostAliases are not supported on hostNetwork Pods. The feature has been supported since v1.8. (kubernetes/kubernetes#122422, @neolit123) [SIG API Machinery and Apps]
Promote AdmissionWebhookMatchConditions to GA. The feature is now stable and the feature gate is now locked to default. (kubernetes/kubernetes#123560, @ivelichkovich) [SIG API Machinery and Testing]
Structured Authentication Configuration now supports DiscoveryURL.
discoveryURL if specified, overrides the URL used to fetch discovery information.
This is for scenarios where the well-known and jwks endpoints are hosted at a different
location than the issuer (such as locally in the cluster). (kubernetes/kubernetes#123527, @aramase) [SIG API Machinery, Auth and Testing]
Support Recursive Read-only (RRO) mounts (KEP-3857) (kubernetes/kubernetes#123180, @AkihiroSuda) [SIG API Machinery, Apps, Node and Testing]
The StructuredAuthenticationConfiguration feature is now beta and enabled by default. (kubernetes/kubernetes#123719, @enj) [SIG API Machinery and Auth]
The StorageVersionMigration API, which was previously available as a Custom Resource Definition (CRD), is now a built-in API in Kubernetes. (kubernetes/kubernetes#123344, @nilekhc) [SIG API Machinery, Apps, Auth, CLI and Testing]
The kubernetes repo now uses Go workspaces. This should not impact end users at all, but does have impact for developers of downstream projects. Switching to workspaces caused some breaking changes in the flags to the various k8s.io/code-generator tools. Downstream consumers should look at staging/src/k8s.io/code-generator/kube_codegen.sh to see the changes. (kubernetes/kubernetes#123529, @thockin) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing]
ValidatingAdmissionPolicy is promoted to GA and will be enabled by default. (kubernetes/kubernetes#123405, @cici37) [SIG API Machinery, Apps, Auth and Testing]
When configuring a JWT authenticator:
If username.expression uses ‘claims.email’, then ‘claims.email_verified’ must be used in
username.expression or extra[].valueExpression or claimValidationRules[].expression.
An example claim validation rule expression that matches the validation automatically
applied when username.claim is set to ‘email’ is ‘claims.?email_verified.orValue(true)’. (kubernetes/kubernetes#123737, @enj) [SIG API Machinery and Auth]
Added a CBOR implementation of runtime.Serializer. Until CBOR graduates to Alpha, API servers will refuse to start if configured with CBOR support. (kubernetes/kubernetes#122881, @benluddy) [SIG API Machinery]
Added audienceMatchPolicy field to AuthenticationConfiguration and support for configuring multiple audiences.
- The “audienceMatchPolicy” can be empty (or unset) when a single audience is specified in the “audiences” field.
- The “audienceMatchPolicy” must be set to “MatchAny” when multiple audiences are specified in the “audiences” field. (kubernetes/kubernetes#123165, @aramase) [SIG API Machinery, Auth and Testing]
Contextual logging is now beta and enabled by default. (kubernetes/kubernetes#122589, @pohly) [SIG Instrumentation]
Cri-api: KEP-3857: Recursive Read-only (RRO) mounts (kubernetes/kubernetes#123272, @AkihiroSuda) [SIG Node]
Enabled a mechanism for concurrent log rotatation via kubelet using a configuration entity of containerLogMaxWorkers which controls the maximum number of concurrent rotation that can be performed and an interval configuration of containerLogMonitorInterval that can aid is configuring the monitoring duration to best suite your cluster’s log generation standards. (kubernetes/kubernetes#114301, @harshanarayana) [SIG API Machinery, Node and Testing]
Text logging in Kubernetes components now uses textlogger. The same split streams of info and error log entries with buffering of info entries is now also supported for text output (off by default, alpha feature). Previously, this was only supported for JSON. Performance is better also without split streams. (kubernetes/kubernetes#114672, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
This change adds the following CLI option for kube-controller-manager:
Updated an audit annotation key used by the …/serviceaccounts/<name>/token resource handler.
The annotation used to persist the issued credential identifier is now authentication.kubernetes.io/issued-credential-id. (kubernetes/kubernetes#123098, @munnerz) [SIG Auth]
Add CEL library for IP Addresses and CIDRs. This will not be available for use until 1.31. (kubernetes/kubernetes#121912, @JoelSpeed) [SIG API Machinery]
Added to MutableFeatureGate the ability to override the default setting of feature gates, to allow default-enabling a feature on a component-by-component basis instead of for all affected components simultaneously. (kubernetes/kubernetes#122647, @benluddy) [SIG API Machinery and Cluster Lifecycle]
Adds a rule on the kube_codegen tool to ignore vendor folder during the code generation. (kubernetes/kubernetes#122729, @jparrill) [SIG API Machinery and Cluster Lifecycle]
Allow users to mutate FSGroupPolicy and PodInfoOnMount in CSIDriver.Spec (kubernetes/kubernetes#116209, @haoruan) [SIG API Machinery, Storage and Testing]
Client-go events: NewEventBroadcasterAdapterWithContext should be used instead of NewEventBroadcasterAdapter if the goal is to support contextual logging. (kubernetes/kubernetes#122142, @pohly) [SIG API Machinery, Instrumentation and Scheduling]
Fixes accidental enablement of the new alpha optionalOldSelf API field in CustomResourceDefinition validation rules, which should only be allowed to be set when the CRDValidationRatcheting feature gate is enabled. (kubernetes/kubernetes#122329, @jpbetz) [SIG API Machinery]
Implement prescore extension point for volumeBinding plugin. Return skip if it doesn’t do anything in Score. (kubernetes/kubernetes#115768, @AxeZhan) [SIG Scheduling, Storage and Testing]
Resource.k8s.io/ResourceClaim (alpha API): the strategic merge patch strategy for the status.reservedFor array was changed such that a strategic-merge-patch can add individual entries. This breaks clients using strategic merge patch to update status which rely on the previous behavior (replacing the entire array). (kubernetes/kubernetes#122276, @pohly) [SIG API Machinery]
When scheduling a mixture of pods using ResourceClaims and others which don’t, scheduling a pod with ResourceClaims impacts scheduling latency less. (kubernetes/kubernetes#121876, @pohly) [SIG API Machinery, Node, Scheduling and Testing]
‘kube-apiserver: adds --authentication-config flag for reading AuthenticationConfiguration
files. --authentication-config flag is mutually exclusive with the existing --oidc-*
flags.’ (kubernetes/kubernetes#119142, @aramase)
‘kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta3
is removed in v1.29. Migrated kube-scheduler configuration files to kubescheduler.config.k8s.io/v1.’ (kubernetes/kubernetes#119994, @SataQiu)
A new sleep action for the PreStop lifecycle hook was added, allowing containers to pause for a specified duration before termination. (kubernetes/kubernetes#119026, @AxeZhan)
Added CEL expressions to v1alpha1 AuthenticationConfiguration. (kubernetes/kubernetes#121078, @aramase)
Added Windows support for InPlace Pod Vertical Scaling feature. (kubernetes/kubernetes#112599, @fabi200123) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
Added ImageMaximumGCAge field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it’s garbage collected. (kubernetes/kubernetes#121275, @haircommander)
Added UserNamespacesPodSecurityStandards feature gate to enable user namespace support for Pod Security Standards.
Enabling this feature will modify all Pod Security Standard rules to allow setting: spec[.*].securityContext.[runAsNonRoot,runAsUser].
This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
The feature gate will not graduate or be enabled by default in future Kubernetes releases. (kubernetes/kubernetes#118760, @saschagrunert) [SIG API Machinery, Auth, Node and Release]
Added optionalOldSelf to x-kubernetes-validations to support ratcheting CRD schema constraints. (kubernetes/kubernetes#121034, @alexzielenski)
Added a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses. (kubernetes/kubernetes#116516, @aojea)
Added a new ipMode field to the .status of Services where type is set to LoadBalancer.
The new field is behind the LoadBalancerIPMode feature gate. (kubernetes/kubernetes#119937, @RyanAoh) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
Added options for configuring nf_conntrack_udp_timeout, and nf_conntrack_udp_timeout_stream variables of netfilter conntrack subsystem. (kubernetes/kubernetes#120808, @aroradaman)
Added support for CEL expressions to v1alpha1 AuthorizationConfiguration webhook matchConditions. (kubernetes/kubernetes#121223, @ritazh)
Added support for projecting certificates.k8s.io/v1alpha1 ClusterTrustBundle objects into pods. (kubernetes/kubernetes#113374, @ahmedtd)
Added the DisableNodeKubeProxyVersion feature gate. If DisableNodeKubeProxyVersion is enabled, the kubeProxyVersion field is not set. (kubernetes/kubernetes#120954, @HirazawaUi)
Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. (kubernetes/kubernetes#119800, @jpbetz) [SIG API Machinery, Auth and Cloud Provider]
Fixed the API comments for the Job Ready field in status. (kubernetes/kubernetes#121765, @mimowo)
Fixed the API comments for the FailIndex Job pod failure policy action. (kubernetes/kubernetes#121764, @mimowo)
Go API: the ResourceRequirements struct was replaced with VolumeResourceRequirements for use with volumes. (kubernetes/kubernetes#118653, @pohly)
Graduated Job BackoffLimitPerIndex feature to beta. (kubernetes/kubernetes#121356, @mimowo)
Marked the onPodConditions field as optional in Job’s pod failure policy. (kubernetes/kubernetes#120204, @mimowo)
Promoted PodReadyToStartContainers condition to beta. (kubernetes/kubernetes#119659, @kannon92)
The flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema and PriorityLevelConfiguration APIs has been promoted to flowcontrol.apiserver.k8s.io/v1, with the following changes:
PriorityLevelConfiguration: the .spec.limited.nominalConcurrencyShares field defaults to 30 only if the field is omitted (v1beta3 also defaulted an explicit 0 value to 30). Specifying an explicit 0 value is not allowed in the v1 version in v1.29 to ensure compatibility with v1.28 API servers. In v1.30, explicit 0 values will be allowed in this field in the v1 API.
The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the v1 APIs. Transition clients and manifests to use the v1 APIs before upgrading to v1.32. (kubernetes/kubernetes#121089, @tkashem)
The kube-proxy command-line documentation was updated to clarify that
--bind-address does not actually have anything to do with binding to an
address, and you probably don’t actually want to be using it. (kubernetes/kubernetes#120274, @danwinship)
The kube-scheduler selectorSpread plugin has been removed, please use the podTopologySpread plugin instead. (kubernetes/kubernetes#117720, @kerthcet)
The matchLabelKeys/mismatchLabelKeys feature is introduced to the hard/soft PodAffinity/PodAntiAffinity. (kubernetes/kubernetes#116065, @sanposhiho)
When updating a CRD, per-expression cost limit check are now skipped for x-kubernetes-validations rules of versions that are not mutated. (kubernetes/kubernetes#121460, @jiahuif)
CSINodeExpandSecret feature has been promoted to GA in this release and is enabled
by default. The CSI drivers can make use of the secretRef values passed in NodeExpansion
request optionally sent by the CSI Client from this release onwards. (kubernetes/kubernetes#121303, @humblec)
NodeStageVolume calls will now be retried if the CSI node driver is not running. (kubernetes/kubernetes#120330, @rohitssingh)
PersistentVolumeLastPhaseTransitionTime is now beta and enabled by default. (kubernetes/kubernetes#120627, @RomanBednar)
ValidatingAdmissionPolicy type checking now supports CRDs and API extensions types. (kubernetes/kubernetes#119109, @jiahuif)
kube-apiserver: added --authorization-config flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration object. The --authorization-config flag is mutually exclusive with --authorization-modes and --authorization-webhook-* flags. The alpha StructuredAuthorizationConfiguration feature flag must be enabled for --authorization-config to be specified. (kubernetes/kubernetes#120154, @palnabarun)
kube-proxy now has a new nftables-based mode, available by running
`kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables`
This is currently an alpha-level feature and while it probably will not
eat your data, it may nibble at it a bit. (It passes e2e testing but has
not yet seen real-world use.)
At this point it should be functionally mostly identical to the iptables
mode, except that it does not (and will not) support Service NodePorts on
127.0.0.1. (Also note that there are currently no command-line arguments
for the nftables-specific config; you will need to use a config file if
you want to set the equivalent of any of the --iptables-xxx options.)
As this code is still very new, it has not been heavily optimized yet;
while it is expected to eventually have better performance than the
iptables backend, very little performance testing has been done so far. (kubernetes/kubernetes#121046, @danwinship)
kube-proxy: Added an option/flag for configuring the nf_conntrack_tcp_be_liberal sysctl (in the kernel’s netfilter conntrack subsystem). When enabled, kube-proxy will not install the DROP rule for invalid conntrack states, which currently breaks users of asymmetric routing. (kubernetes/kubernetes#120354, @aroradaman)
Added support for projecting certificates.k8s.io/v1alpha1 ClusterTrustBundle objects into pods. (kubernetes/kubernetes#113374, @ahmedtd) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
Adds optionalOldSelf to x-kubernetes-validations to support ratcheting CRD schema constraints (kubernetes/kubernetes#121034, @alexzielenski) [SIG API Machinery]
Fix API comment for the Job Ready field in status (kubernetes/kubernetes#121765, @mimowo) [SIG API Machinery and Apps]
Fix API comments for the FailIndex Job pod failure policy action. (kubernetes/kubernetes#121764, @mimowo) [SIG API Machinery and Apps]
A new sleep action for the PreStop lifecycle hook is added, allowing containers to pause for a specified duration before termination. (kubernetes/kubernetes#119026, @AxeZhan) [SIG API Machinery, Apps, Node and Testing]
Add ImageMaximumGCAge field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it’s garbage collected. (kubernetes/kubernetes#121275, @haircommander) [SIG API Machinery and Node]
Add a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses (kubernetes/kubernetes#116516, @aojea) [SIG API Machinery, Apps, Auth, CLI, Network and Testing]
Add the DisableNodeKubeProxyVersion feature gate. If DisableNodeKubeProxyVersion is enabled, the kubeProxyVersion field is not set. (kubernetes/kubernetes#120954, @HirazawaUi) [SIG API Machinery, Apps and Node]
Added Windows support for InPlace Pod Vertical Scaling feature. (kubernetes/kubernetes#112599, @fabi200123) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
Added UserNamespacesPodSecurityStandards feature gate to enable user namespace support for Pod Security Standards.
Enabling this feature will modify all Pod Security Standard rules to allow setting: spec[.*].securityContext.[runAsNonRoot,runAsUser].
This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
The feature gate will not graduate or be enabled by default in future Kubernetes releases. (kubernetes/kubernetes#118760, @saschagrunert) [SIG API Machinery, Auth, Node and Release]
Added options for configuring nf_conntrack_udp_timeout, and nf_conntrack_udp_timeout_stream variables of netfilter conntrack subsystem. (kubernetes/kubernetes#120808, @aroradaman) [SIG API Machinery and Network]
Adds CEL expressions to v1alpha1 AuthenticationConfiguration. (kubernetes/kubernetes#121078, @aramase) [SIG API Machinery, Auth and Testing]
Adds support for CEL expressions to v1alpha1 AuthorizationConfiguration webhook matchConditions. (kubernetes/kubernetes#121223, @ritazh) [SIG API Machinery and Auth]
CSINodeExpandSecret feature has been promoted to GA in this release and enabled by default. The CSI drivers can make use of the secretRef values passed in NodeExpansion request optionally sent by the CSI Client from this release onwards. (kubernetes/kubernetes#121303, @humblec) [SIG API Machinery, Apps and Storage]
Graduate Job BackoffLimitPerIndex feature to Beta (kubernetes/kubernetes#121356, @mimowo) [SIG Apps]
Kube-apiserver: adds –authorization-config flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration object. –authorization-config flag is mutually exclusive with –authorization-modes and –authorization-webhook-* flags. The alpha StructuredAuthorizationConfiguration feature flag must be enabled for –authorization-config to be specified. (kubernetes/kubernetes#120154, @palnabarun) [SIG API Machinery, Auth and Testing]
Kube-proxy now has a new nftables-based mode, available by running
kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables
This is currently an alpha-level feature and while it probably will not
eat your data, it may nibble at it a bit. (It passes e2e testing but has
not yet seen real-world use.)
At this point it should be functionally mostly identical to the iptables
mode, except that it does not (and will not) support Service NodePorts on
127.0.0.1. (Also note that there are currently no command-line arguments
for the nftables-specific config; you will need to use a config file if
you want to set the equivalent of any of the --iptables-xxx options.)
As this code is still very new, it has not been heavily optimized yet;
while it is expected to eventually have better performance than the
iptables backend, very little performance testing has been done so far. (kubernetes/kubernetes#121046, @danwinship) [SIG API Machinery and Network]
Kube-proxy: Added an option/flag for configuring the nf_conntrack_tcp_be_liberal sysctl (in the kernel’s netfilter conntrack subsystem). When enabled, kube-proxy will not install the DROP rule for invalid conntrack states, which currently breaks users of asymmetric routing. (kubernetes/kubernetes#120354, @aroradaman) [SIG API Machinery and Network]
PersistentVolumeLastPhaseTransitionTime is now beta, enabled by default. (kubernetes/kubernetes#120627, @RomanBednar) [SIG Storage]
Promote PodReadyToStartContainers condition to beta. (kubernetes/kubernetes#119659, @kannon92) [SIG Node and Testing]
The flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema and PriorityLevelConfiguration APIs has been promoted to flowcontrol.apiserver.k8s.io/v1, with the following changes:
- PriorityLevelConfiguration: the
.spec.limited.nominalConcurrencyShares field defaults to 30 only if the field is omitted (v1beta3 also defaulted an explicit 0 value to 30). Specifying an explicit 0 value is not allowed in the v1 version in v1.29 to ensure compatibility with 1.28 API servers. In v1.30, explicit 0 values will be allowed in this field in the v1 API.
The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the v1 APIs. Transition clients and manifests to use the v1 APIs before upgrading to v1.32. (kubernetes/kubernetes#121089, @tkashem) [SIG API Machinery and Testing]
The kube-proxy command-line documentation was updated to clarify that
--bind-address does not actually have anything to do with binding to an
address, and you probably don’t actually want to be using it. (kubernetes/kubernetes#120274, @danwinship) [SIG Network]
The matchLabelKeys/mismatchLabelKeys feature is introduced to the hard/soft PodAffinity/PodAntiAffinity. (kubernetes/kubernetes#116065, @sanposhiho) [SIG API Machinery, Apps, Cloud Provider, Scheduling and Testing]
ValidatingAdmissionPolicy Type Checking now supports CRDs and API extensions types. (kubernetes/kubernetes#119109, @jiahuif) [SIG API Machinery, Apps, Auth and Testing]
When updating a CRD, per-expression cost limit check is skipped for x-kubernetes-validations rules of versions that are not mutated. (kubernetes/kubernetes#121460, @jiahuif) [SIG API Machinery]
Added a new ipMode field to the .status of Services where type is set to LoadBalancer.
The new field is behind the LoadBalancerIPMode feature gate. (kubernetes/kubernetes#119937, @RyanAoh) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. (kubernetes/kubernetes#119800, @jpbetz) [SIG API Machinery, Auth and Cloud Provider]
Go API: the ResourceRequirements struct needs to be replaced with VolumeResourceRequirements for use with volumes. (kubernetes/kubernetes#118653, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling, Storage and Testing]
Kube-apiserver: adds –authentication-config flag for reading AuthenticationConfiguration files. –authentication-config flag is mutually exclusive with the existing –oidc-* flags. (kubernetes/kubernetes#119142, @aramase) [SIG API Machinery, Auth and Testing]
Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta3 is removed in v1.29. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. (kubernetes/kubernetes#119994, @SataQiu) [SIG Scheduling and Testing]
Mark the onPodConditions field as optional in Job’s pod failure policy. (kubernetes/kubernetes#120204, @mimowo) [SIG API Machinery and Apps]
Retry NodeStageVolume calls if CSI node driver is not running (kubernetes/kubernetes#120330, @rohitssingh) [SIG Apps, Storage and Testing]
The kube-scheduler selectorSpread plugin has been removed, please use the podTopologySpread plugin instead. (kubernetes/kubernetes#117720, @kerthcet) [SIG Scheduling]
Added error handling for seccomp localhost configurations that do not properly set a localhostProfile (kubernetes/kubernetes#117020, @cji) [SIG API Machinery and Node]
Fixed an issue where kubelet does not set case-insensitive headers for http probes. (#117182, @dddddai) (kubernetes/kubernetes#117324, @dddddai) [SIG API Machinery, Apps and Node]
Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta (kubernetes/kubernetes#117815, @kerthcet) [SIG Apps]
A fix in the resource.k8s.io/v1alpha1/ResourceClaim API avoids harmless (?) “.status.reservedFor: element 0: associative list without keys has an element that’s a map type” errors in the apiserver. Validation now rejects the incorrect reuse of the same UID in different entries. (kubernetes/kubernetes#115354, @pohly)
A terminating pod on a node that is not caused by preemption no longer prevents kube-scheduler from preempting pods on that node
API: resource.k8s.io/v1alpha1.PodScheduling was renamed to resource.k8s.io/v1alpha2.PodSchedulingContext. (kubernetes/kubernetes#116556, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Scheduling and Testing]
Added CEL runtime cost calculation into ValidatingAdmissionPolicy, matching the evaluation cost
restrictions that already apply to CustomResourceDefinition.
If rule evaluation uses more compute than the limit, the API server aborts the evaluation and the
admission check that was being performed is aborted; the failurePolicy for the ValidatingAdmissionPolicy
determines the outcome. (kubernetes/kubernetes#115747, @cici37)
Added auditAnnotations to ValidatingAdmissionPolicy, enabling CEL to be used to add audit annotations to request audit events.
Added validationActions to ValidatingAdmissionPolicyBinding, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions. (kubernetes/kubernetes#115973, @jpbetz)
Added messageExpression field to ValidationRule. (kubernetes/kubernetes#115969, @DangerOnTheRanger)
Added messageExpression to ValidatingAdmissionPolicy, to set custom failure message via CEL expression. (kubernetes/kubernetes#116397, @jiahuif) [SIG API Machinery]
Added a new IPAddress object kind
- Added a new ClusterIP allocator. The new allocator removes previous Service CIDR block size limitations for IPv4, and limits IPv6 size to a /64 (kubernetes/kubernetes#115075, @aojea) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Network and Testing]
Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (kubernetes/kubernetes#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]
Added authorization check support to the CEL expressions of ValidatingAdmissionPolicy via a authorizer
variable with expressions. The new variable provides a builder that allows expressions such authorizer.group('').resource('pods').check('create').allowed(). (kubernetes/kubernetes#116054, @jpbetz) [SIG API Machinery and Testing]
Added matchConditions field to ValidatingAdmissionPolicy and enabled support for CEL based custom match criteria. (kubernetes/kubernetes#116350, @maxsmythe)
Added new option to the InterPodAffinity scheduler plugin to ignore existing
podspreferred inter-pod affinities if the incoming pod has no preferred inter-pod affinities. This option can be used as an optimization for higher scheduling throughput (at the cost of an occasional pod being scheduled non-optimally/violating existing pods preferred inter-pod affinities). To enable this scheduler option, set theInterPodAffinityscheduler plugin argignorePreferredTermsOfExistingPods: true` (kubernetes/kubernetes#114393, @danielvegamyhre)
Added the MatchConditions field to ValidatingWebhookConfiguration and MutatingWebhookConfiguration for the v1beta and v1 apis.
The AdmissionWebhookMatchConditions featuregate is now in Alpha (kubernetes/kubernetes#116261, @ivelichkovich) [SIG API Machinery and Testing]
Added validation to ensure that if service.kubernetes.io/topology-aware-hints and service.kubernetes.io/topology-mode annotations are both set, they are set to the same value.Also Added deprecation warning if service.kubernetes.io/topology-aware-hints annotation is used. (kubernetes/kubernetes#116612, @robscott)
Added warnings about workload resources (Pods, ReplicaSets, Deployments, Jobs, CronJobs, or ReplicationControllers) whose names are not valid DNS labels. (kubernetes/kubernetes#114412, @thockin)
Adds feature gate NodeLogQuery which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node. (kubernetes/kubernetes#96120, @LorbusChris)
Api: validation of a PodSpec now rejects invalid ResourceClaim and ResourceClaimTemplate names. For a pod, the name generated for the ResourceClaim when using a template also must be valid. (kubernetes/kubernetes#116576, @pohly)
Bump default API QPS limits for Kubelet. (kubernetes/kubernetes#116121, @wojtek-t)
Enabled the StatefulSetStartOrdinal feature gate in beta (kubernetes/kubernetes#115260, @pwschuurman)
Enabled usage of kube-proxy, kube-scheduler and kubelet HTTP APIs for changing the logging
verbosity at runtime for JSON output. (kubernetes/kubernetes#114609, @pohly)
Encryption of API Server at rest configuration now allows the use of wildcards in the list of resources. For example, . can be used to encrypt all resources, including all current and future custom resources. (kubernetes/kubernetes#115149, @nilekhc)
Extended the kubelet’s PodResources API to include resources allocated in ResourceClaims via DynamicResourceAllocation. Additionally, added a new Get() method to query a specific pod for its resources. (kubernetes/kubernetes#115847, @moshe010) [SIG Node]
Forbid to set matchLabelKeys when labelSelector is not set in topologySpreadConstraints (kubernetes/kubernetes#116535, @denkensk)
GCE does not support LoadBalancer Services with ports with different protocols (TCP and UDP) (kubernetes/kubernetes#115966, @aojea) [SIG Apps and Cloud Provider]
GRPC probes are now a GA feature. GRPCContainerProbe feature gate was locked to default value and will be removed in v1.29. If you were setting this feature gate explicitly, please remove it now. (kubernetes/kubernetes#116233, @SergeyKanzhelev)
Graduated Kubelet Topology Manager to GA. (kubernetes/kubernetes#116093, @swatisehgal)
Graduated KubeletTracing to beta, which means that the feature gate is now enabled by default. (kubernetes/kubernetes#115750, @saschagrunert)
Graduated seccomp profile defaulting to GA.
Set the kubelet --seccomp-default flag or seccompDefault kubelet configuration field to true to make pods on that node default to using the RuntimeDefault seccomp profile.
Enabling seccomp for your workload can have a negative performance impact depending on the kernel and container runtime version in use.
Guidance for identifying and mitigating those issues is outlined in the Kubernetes seccomp tutorial. (kubernetes/kubernetes#115719, @saschagrunert) [SIG API Machinery, Node, Storage and Testing]
Graduated the container resource metrics feature on HPA to beta. (kubernetes/kubernetes#116046, @sanposhiho)
Implemented API streaming for the watch-cache
When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic “Bookmark” after which the server continues streaming events. (kubernetes/kubernetes#110960, @p0lyn0mial)
Introduced API for streaming.
Added SendInitialEvents field to the ListOptions. When the new option is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic “Bookmark” after which the server continues streaming events. (kubernetes/kubernetes#115402, @p0lyn0mial)
Introduced a breaking change to the resource.k8s.io API in its AllocationResult struct. This change allows a kubelet plugin for the DynamicResourceAllocation feature to service allocations from multiple resource driver controllers. (kubernetes/kubernetes#116332, @klueska)
Introduces new alpha functionality to the reflector, allowing user to enable API streaming.
To activate this feature, users can set the ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environmental variable.
It is important to note that the server must support streaming for this feature to function properly.
If streaming is not supported by the server, the reflector will revert to the previous method
of obtaining data through LIST/WATCH semantics. (kubernetes/kubernetes#110772, @p0lyn0mial) [SIG API Machinery]
K8s.io/client-go/tools/record.EventBroadcaster: after Shutdown() is called, the broadcaster now gives up immediately after a failure to write an event to a sink. Previously it tried multiple times for 12 seconds in a goroutine. (kubernetes/kubernetes#115514, @pohly) [SIG API Machinery]
K8s.io/component-base/logs: usage of the pflag values in a normal Go flag set led to panics when printing the help message (kubernetes/kubernetes#114680, @pohly) [SIG Instrumentation]
Kubeadm: explicitly set priority for static pods with priorityClassName: system-node-critical (kubernetes/kubernetes#114338, @champtar) [SIG Cluster Lifecycle]
Kubelet: a “maxParallelImagePulls” field can now be specified in the kubelet configuration file to control how many image pulls the kubelet can perform in parallel. (kubernetes/kubernetes#115220, @ruiwen-zhao) [SIG API Machinery, Node and Scalability]
Kubelet: changed MemoryThrottlingFactor default value to 0.9 and formulas to calculate memory.high (kubernetes/kubernetes#115371, @pacoxu)
Kubernetes components that perform leader election now only support using Leases for this. (kubernetes/kubernetes#114055, @aimuz)
Migrated the DaemonSet controller (within kube-controller-manager) to use contextual logging (kubernetes/kubernetes#113622, @249043822)
New service.kubernetes.io/topology-mode annotation has been introduced as a replacement for the service.kubernetes.io/topology-aware-hints annotation.
service.kubernetes.io/topology-aware-hints annotation has been deprecated.- kube-proxy now accepts any value that is not “disabled” for these annotations, enabling custom implementation-specific and/or future built-in heuristics to be used. (kubernetes/kubernetes#116522, @robscott) [SIG Apps, Network and Testing]
Pods owned by a Job now uses the labels batch.kubernetes.io/job-name and batch.kubernetes.io/controller-uid.
The legacy labels job-name and controller-uid are still added for compatibility. (kubernetes/kubernetes#114930, @kannon92)
Promoted CronJobTimeZone feature to GA (kubernetes/kubernetes#115904, @soltysh)
Promoted SelfSubjectReview to Beta (kubernetes/kubernetes#116274, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]
Relaxed API validation to allow pod node selector to be mutable for gated pods (additions only, no deletions or mutations). (kubernetes/kubernetes#116161, @danielvegamyhre)
Remove kubernetes.io/grpc standard appProtocol (kubernetes/kubernetes#116866, @LiorLieberman) [SIG API Machinery and Apps]
Remove deprecated --enable-taint-manager and --pod-eviction-timeout CLI (kubernetes/kubernetes#115840, @atosatto)
Removed support for the v1alpha1 kubeletplugin API of DynamicResourceManagement. All plugins must be updated to v1alpha2 in order to function properly. (kubernetes/kubernetes#116558, @klueska)
The API server now re-uses data encryption keys while the kms v2 plugin key ID is stable. Data encryption keys are still randomly generated on server start but an atomic counter is used to prevent nonce collisions. (kubernetes/kubernetes#116155, @enj)
The PodDisruptionBudget spec.unhealthyPodEvictionPolicy field has graduated to beta and is enabled by default. On servers with the feature enabled, this field may be set to AlwaysAllow to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. (kubernetes/kubernetes#115363, @ravisantoshgudimetla) [SIG Apps, Auth and Node]
The DownwardAPIHugePages kubelet feature graduated to stable / GA. (kubernetes/kubernetes#115721, @saschagrunert) [SIG Apps and Node]
The following feature gates for volume expansion GA features have now been removed and must no longer be referenced in --feature-gates flags: ExpandCSIVolumes, ExpandInUsePersistentVolumes, ExpandPersistentVolumes (kubernetes/kubernetes#113942, @mengjiao-liu)
The list-type of the alpha resourceClaims field introduced to Pods in 1.26.0 was modified from set to map, resolving an incompatibility with use of this schema in CustomResourceDefinitions and with server-side apply. (kubernetes/kubernetes#114585, @JoelSpeed)
Updated API reference for Requests, specifying they must not exceed limits (kubernetes/kubernetes#115434, @ehashman)
Updated KMSv2 to beta (kubernetes/kubernetes#115123, @aramase)
Updated: Redefine AppProtocol field description and add new standard values (kubernetes/kubernetes#115433, @LiorLieberman) [SIG API Machinery, Apps and Network]
/metrics/slis is now available for control plane components allowing you to scrape health check metrics. (kubernetes/kubernetes#114997, @Richabanker)
APIServerTracing feature gate is now enabled by default. Tracing in the API
Server is still disabled by default, and requires a config file to enable. (kubernetes/kubernetes#116144, @dashpole)
NodeResourceFit and NodeResourcesBalancedAllocation implement the PreScore
extension point for a more performant calculation. (kubernetes/kubernetes#115655, @tangwz)
PodSchedulingReadiness is graduated to beta. (kubernetes/kubernetes#115815, @Huang-Wei)
PodSpec.Container.Resources became mutable for CPU and memory resource types.
PodSpec.Container.ResizePolicy (new object) gives users control over how their containers are resized.PodStatus.Resize status describes the state of a requested Pod resize.PodStatus.ResourcesAllocated describes node resources allocated to Pod.PodStatus.Resources describes node resources applied to running containers by CRI.UpdateContainerResources CRI API now supports both Linux and Windows. (kubernetes/kubernetes#102884, @vinaykul)
SELinuxMountReadWriteOncePod graduated to Beta. (kubernetes/kubernetes#116425, @jsafrane)
StatefulSetAutoDeletePVC feature gate promoted to beta. (kubernetes/kubernetes#116501, @mattcary)
StatefulSet names must be DNS labels, rather than subdomains. Any StatefulSet
which took advantage of subdomain validation (by having dots in the name) can’t
possibly have worked, because we eventually set pod.spec.hostname from the StatefulSetName,
and that is validated as a DNS label. (kubernetes/kubernetes#114172, @thockin)
ValidatingAdmissionPolicy now provides a status field that contains results of type checking the validation expression.
The type checking is fully informational, and the behavior of the policy is unchanged. (kubernetes/kubernetes#115668, @jiahuif)
cacheSize field in EncryptionConfiguration is not supported for KMSv2 provider (kubernetes/kubernetes#113121, @aramase)
k8s.io/component-base/logs now also supports adding command line flags to a flag.FlagSet. (kubernetes/kubernetes#114731, @pohly)
kubelet: migrated --container-runtime-endpoint and --image-service-endpoint
to kubelet config (kubernetes/kubernetes#112136, @pacoxu)
resource.k8s.io/v1alpha1 was replaced with resource.k8s.io/v1alpha2. Before
upgrading a cluster, all objects in resource.k8s.io/v1alpha1 (ResourceClaim, ResourceClaimTemplate,
ResourceClass, PodScheduling) must be deleted. The changes are internal, so
YAML files which create pods and resource claims don’t need changes except for
the newer apiVersion. (kubernetes/kubernetes#116299, @pohly)
volumes: resource.claims is now cleared for PVC specs during create or update of a pod spec with inline PVC template or of a PVC because it has no effect. (kubernetes/kubernetes#115928, @pohly)
Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (kubernetes/kubernetes#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]
Remove kubernetes.io/grpc standard appProtocol (kubernetes/kubernetes#116866, @LiorLieberman) [SIG API Machinery and Apps]
API: resource.k8s.io/v1alpha1.PodScheduling was renamed to resource.k8s.io/v1alpha2.PodSchedulingContext. (kubernetes/kubernetes#116556, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Scheduling and Testing]
APIServerTracing feature gate is now enabled by default. Tracing in the API Server is still disabled by default, and requires a config file to enable. (kubernetes/kubernetes#116144, @dashpole) [SIG API Machinery and Testing]
Added CEL runtime cost calculation into ValidatingAdmissionPolicy, matching the evaluation cost
restrictions that already apply to CustomResourceDefinition.
If rule evaluation uses more compute than the limit, the API server aborts the evaluation and the
admission check that was being performed is aborted; the failurePolicy for the ValidatingAdmissionPolicy
determines the outcome. (kubernetes/kubernetes#115747, @cici37) [SIG API Machinery]
Added messageExpression to ValidatingAdmissionPolicy, to set custom failure message via CEL expression. (kubernetes/kubernetes#116397, @jiahuif) [SIG API Machinery]
Added a new IPAddress object kind
- Added a new ClusterIP allocator. The new allocator removes previous Service CIDR block size limitations for IPv4, and limits IPv6 size to a /64 (kubernetes/kubernetes#115075, @aojea) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Network and Testing]
Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (kubernetes/kubernetes#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]
Added authorization check support to the CEL expressions of ValidatingAdmissionPolicy via a authorizer
variable with expressions. The new variable provides a builder that allows expressions such authorizer.group('').resource('pods').check('create').allowed(). (kubernetes/kubernetes#116054, @jpbetz) [SIG API Machinery and Testing]
Added matchConditions field to ValidatingAdmissionPolicy, enabled support for CEL based custom match criteria. (kubernetes/kubernetes#116350, @maxsmythe) [SIG API Machinery and Testing]
Added messageExpression field to ValidationRule. (#115969, @DangerOnTheRanger) (kubernetes/kubernetes#115969, @DangerOnTheRanger) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node and Testing]
Added the MatchConditions field to ValidatingWebhookConfiguration and MutatingWebhookConfiguration for the v1beta and v1 apis.
The AdmissionWebhookMatchConditions featuregate is now in Alpha (kubernetes/kubernetes#116261, @ivelichkovich) [SIG API Machinery and Testing]
Added validation to ensure that if service.kubernetes.io/topology-aware-hints and service.kubernetes.io/topology-mode annotations are both set, they are set to the same value.
Adds auditAnnotations to ValidatingAdmissionPolicy, enabling CEL to be used to add audit annotations to request audit events.
Adds validationActions to ValidatingAdmissionPolicyBinding, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions. (kubernetes/kubernetes#115973, @jpbetz) [SIG API Machinery and Testing]
Adds feature gate NodeLogQuery which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node. (kubernetes/kubernetes#96120, @LorbusChris) [SIG API Machinery, Apps, CLI, Node, Testing and Windows]
Api: validation of a PodSpec now rejects invalid ResourceClaim and ResourceClaimTemplate names. For a pod, the name generated for the ResourceClaim when using a template also must be valid. (kubernetes/kubernetes#116576, @pohly) [SIG Apps]
Bump default API QPS limits for Kubelet. (kubernetes/kubernetes#116121, @wojtek-t) [SIG API Machinery and Node]
Enable the “StatefulSetStartOrdinal” feature gate in beta (kubernetes/kubernetes#115260, @pwschuurman) [SIG API Machinery and Apps]
Extended the kubelet’s PodResources API to include resources allocated in ResourceClaims via DynamicResourceAllocation. Additionally, added a new Get() method to query a specific pod for its resources. (kubernetes/kubernetes#115847, @moshe010) [SIG Node]
Forbid to set matchLabelKeys when labelSelector isn’t set in topologySpreadConstraints (kubernetes/kubernetes#116535, @denkensk) [SIG API Machinery, Apps and Scheduling]
GCE does not support LoadBalancer Services with ports with different protocols (TCP and UDP) (kubernetes/kubernetes#115966, @aojea) [SIG Apps and Cloud Provider]
GRPC probes are now a GA feature. GRPCContainerProbe feature gate was locked to default value and will be removed in v1.29. If you were setting this feature gate explicitly, please remove it now. (kubernetes/kubernetes#116233, @SergeyKanzhelev) [SIG API Machinery, Apps and Node]
Graduate Kubelet Topology Manager to GA. (kubernetes/kubernetes#116093, @swatisehgal) [SIG API Machinery, Node and Testing]
Graduate KubeletTracing to beta, which means that the feature gate is now enabled by default. (kubernetes/kubernetes#115750, @saschagrunert) [SIG Instrumentation and Node]
Graduate the container resource metrics feature on HPA to beta. (kubernetes/kubernetes#116046, @sanposhiho) [SIG Autoscaling]
Introduced a breaking change to the resource.k8s.io API in its AllocationResult struct. This change allows a kubelet plugin for the DynamicResourceAllocation feature to service allocations from multiple resource driver controllers. (kubernetes/kubernetes#116332, @klueska) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
Introduces new alpha functionality to the reflector, allowing user to enable API streaming.
To activate this feature, users can set the ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environmental variable.
It is important to note that the server must support streaming for this feature to function properly.
If streaming is not supported by the server, the reflector will revert to the previous method
of obtaining data through LIST/WATCH semantics. (kubernetes/kubernetes#110772, @p0lyn0mial) [SIG API Machinery]
Kubelet: change MemoryThrottlingFactor default value to 0.9 and formulas to calculate memory.high (kubernetes/kubernetes#115371, @pacoxu) [SIG API Machinery, Apps and Node]
Migrated the DaemonSet controller (within `kube-controller-manager) to use contextual logging (kubernetes/kubernetes#113622, @249043822) [SIG API Machinery, Apps, Instrumentation and Testing]
New service.kubernetes.io/topology-mode annotation has been introduced as a replacement for the service.kubernetes.io/topology-aware-hints annotation.
service.kubernetes.io/topology-aware-hints annotation has been deprecated.- kube-proxy now accepts any value that is not “disabled” for these annotations, enabling custom implementation-specific and/or future built-in heuristics to be used. (kubernetes/kubernetes#116522, @robscott) [SIG Apps, Network and Testing]
NodeResourceFit and NodeResourcesBalancedAllocation implement the PreScore extension point for a more performant calculation. (kubernetes/kubernetes#115655, @tangwz) [SIG Scheduling]
Pods owned by a Job will now use the labels batch.kubernetes.io/job-name and batch.kubernetes.io/controller-uid.
The legacy labels job-name and controller-uid are still added for compatibility. (kubernetes/kubernetes#114930, @kannon92) [SIG Apps]
Promote CronJobTimeZone feature to GA (kubernetes/kubernetes#115904, @soltysh) [SIG API Machinery and Apps]
Promoted SelfSubjectReview to Beta (kubernetes/kubernetes#116274, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]
Relax API validation to allow pod node selector to be mutable for gated pods (additions only, no deletions or mutations). (kubernetes/kubernetes#116161, @danielvegamyhre) [SIG Apps, Scheduling and Testing]
Remove deprecated --enable-taint-manager and --pod-eviction-timeout CLI flags (kubernetes/kubernetes#115840, @atosatto) [SIG API Machinery, Apps, Node and Testing]
Resource.k8s.io/v1alpha1 was replaced with resource.k8s.io/v1alpha2. Before upgrading a cluster, all objects in resource.k8s.io/v1alpha1 (ResourceClaim, ResourceClaimTemplate, ResourceClass, PodScheduling) must be deleted. The changes will be internal, so YAML files which create pods and resource claims don’t need changes except for the newer apiVersion. (kubernetes/kubernetes#116299, @pohly) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
SELinuxMountReadWriteOncePod graduated to Beta. (kubernetes/kubernetes#116425, @jsafrane) [SIG Storage and Testing]
StatefulSetAutoDeletePVC feature gate promoted to beta. (kubernetes/kubernetes#116501, @mattcary) [SIG Apps, Auth and Testing]
The API server now re-uses data encryption keys while the kms v2 plugin’s key ID is stable. Data encryption keys are still randomly generated on server start but an atomic counter is used to prevent nonce collisions. (kubernetes/kubernetes#116155, @enj) [SIG API Machinery, Auth and Testing]
The API server’s encryption at rest configuration now allows the use of wildcards in the list of resources. For example, ‘.’ can be used to encrypt all resources, including all current and future custom resources. (kubernetes/kubernetes#115149, @nilekhc) [SIG API Machinery, Auth and Testing]
Update KMSv2 to beta (kubernetes/kubernetes#115123, @aramase) [SIG API Machinery, Auth and Testing]
Updated: Redefine AppProtocol field description and add new standard values (kubernetes/kubernetes#115433, @LiorLieberman) [SIG API Machinery, Apps and Network]
ValidatingAdmissionPolicy now provides a status field that contains results of type checking the validation expression.
The type checking is fully informational, and the behavior of the policy is unchanged. (kubernetes/kubernetes#115668, @jiahuif) [SIG API Machinery, Auth, Cloud Provider and Testing]
We have removed support for the v1alpha1 kubeletplugin API of DynamicResourceManagement. All plugins must update to v1alpha2 in order to function properly going forward. (kubernetes/kubernetes#116558, @klueska) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
Graduated seccomp profile defaulting to GA.
Set the kubelet --seccomp-default flag or seccompDefault kubelet configuration field to true to make pods on that node default to using the RuntimeDefault seccomp profile.
Enabling seccomp for your workload can have a negative performance impact depending on the kernel and container runtime version in use.
Guidance for identifying and mitigating those issues is outlined in the Kubernetes seccomp tutorial. (kubernetes/kubernetes#115719, @saschagrunert) [SIG API Machinery, Node, Storage and Testing]
Implements API for streaming for the watch-cache
When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic “Bookmark” after which the server continues streaming events. (kubernetes/kubernetes#110960, @p0lyn0mial) [SIG API Machinery]
Introduce API for streaming.
Add SendInitialEvents field to the ListOptions. When the new option is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic “Bookmark” after which the server continues streaming events. (kubernetes/kubernetes#115402, @p0lyn0mial) [SIG API Machinery]
Kubelet: a “maxParallelImagePulls” field can now be specified in the kubelet configuration file to control how many image pulls the kubelet can perform in parallel. (kubernetes/kubernetes#115220, @ruiwen-zhao) [SIG API Machinery, Node and Scalability]
PodSchedulingReadiness is graduated to beta. (kubernetes/kubernetes#115815, @Huang-Wei) [SIG API Machinery, Apps, Scheduling and Testing]
In-place resize feature for Kubernetes Pods
- Changed the Pod API so that the
resources defined for containers are mutable for cpu and memory resource types.
- Added
resizePolicy for containers in a pod to allow users control over how their containers are resized.
- Added
allocatedResources field to container status in pod status that describes the node resources allocated to a pod.
- Added
resources field to container status that reports actual resources applied to running containers.
- Added
resize field to pod status that describes the state of a requested pod resize.
For details, see KEPs below. (kubernetes/kubernetes#102884, @vinaykul) [SIG API Machinery, Apps, Instrumentation, Node, Scheduling and Testing]
The PodDisruptionBudget spec.unhealthyPodEvictionPolicy field has graduated to beta and is enabled by default. On servers with the feature enabled, this field may be set to AlwaysAllow to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. (kubernetes/kubernetes#115363, @ravisantoshgudimetla) [SIG Apps, Auth and Node]
The DownwardAPIHugePages kubelet feature graduated to stable / GA. (kubernetes/kubernetes#115721, @saschagrunert) [SIG Apps and Node]
Volumes: resource.claims gets cleared for PVC specs during create or update of a pod spec with inline PVC template or of a PVC because it has no effect. (kubernetes/kubernetes#115928, @pohly) [SIG API Machinery, Apps and Storage]
A fix in the resource.k8s.io/v1alpha1/ResourceClaim API avoids harmless (?) “.status.reservedFor: element 0: associative list without keys has an element that’s a map type” errors in the apiserver. Validation now rejects the incorrect reuse of the same UID in different entries. (kubernetes/kubernetes#115354, @pohly) [SIG API Machinery]
CacheSize field in EncryptionConfiguration is not supported for KMSv2 provider (kubernetes/kubernetes#113121, @aramase) [SIG API Machinery, Auth and Testing]
K8s.io/client-go/tools/record.EventBroadcaster: after Shutdown() is called, the broadcaster now gives up immediately after a failure to write an event to a sink. Previously it tried multiple times for 12 seconds in a goroutine. (kubernetes/kubernetes#115514, @pohly) [SIG API Machinery]
K8s.io/component-base/logs now also supports adding command line flags to a flag.FlagSet. (kubernetes/kubernetes#114731, @pohly) [SIG Architecture]
Update API reference for Requests, specifying they must not exceed limits (kubernetes/kubernetes#115434, @ehashman) [SIG Architecture, Docs and Node]
/metrics/slis is made available for control plane components allowing you to scrape health check metrics. (kubernetes/kubernetes#114997, @Richabanker) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]
A terminating pod on a node that is not caused by preemption won’t prevent kube-scheduler from preempting pods on that node
Added new option to the InterPodAffinity scheduler plugin to ignore existing pods` preferred inter-pod affinities if the incoming pod has no preferred inter-pod affinities. This option can be used as an optimization for higher scheduling throughput (at the cost of an occasional pod being scheduled non-optimally/violating existing pods’ preferred inter-pod affinities). To enable this scheduler option, set the InterPodAffinity scheduler plugin arg “ignorePreferredTermsOfExistingPods: true”. (kubernetes/kubernetes#114393, @danielvegamyhre) [SIG API Machinery and Scheduling]
Added warnings about workload resources (Pods, ReplicaSets, Deployments, Jobs, CronJobs, or ReplicationControllers) whose names are not valid DNS labels. (kubernetes/kubernetes#114412, @thockin) [SIG API Machinery and Apps]
K8s.io/component-base/logs: usage of the pflag values in a normal Go flag set led to panics when printing the help message (kubernetes/kubernetes#114680, @pohly) [SIG Instrumentation]
Kube-proxy, kube-scheduler and kubelet have HTTP APIs for changing the logging verbosity at runtime. This now also works for JSON output. (kubernetes/kubernetes#114609, @pohly) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation and Testing]
Kubeadm: explicitly set priority for static pods with priorityClassName: system-node-critical (kubernetes/kubernetes#114338, @champtar) [SIG Cluster Lifecycle]
Kubelet: migrate “–container-runtime-endpoint” and “–image-service-endpoint” to kubelet config (kubernetes/kubernetes#112136, @pacoxu) [SIG API Machinery, Node and Scalability]
Kubernetes components that perform leader election now only support using Leases for this. (kubernetes/kubernetes#114055, @aimuz) [SIG API Machinery, Cloud Provider and Scheduling]
StatefulSet names must be DNS labels, rather than subdomains. Any StatefulSet which took advantage of subdomain validation (by having dots in the name) can’t possibly have worked, because we eventually set pod.spec.hostname from the StatefulSetName, and that is validated as a DNS label. (kubernetes/kubernetes#114172, @thockin) [SIG Apps]
The following feature gates for volume expansion GA features have been removed and must no longer be referenced in --feature-gates flags: ExpandCSIVolumes, ExpandInUsePersistentVolumes, ExpandPersistentVolumes (kubernetes/kubernetes#113942, @mengjiao-liu) [SIG API Machinery, Apps and Testing]
The list-type of the alpha resourceClaims field introduced to Pods in 1.26.0 was modified from “set” to “map”, resolving an incompatibility with use of this schema in CustomResourceDefinitions and with server-side apply. (kubernetes/kubernetes#114585, @JoelSpeed) [SIG API Machinery]
The list-type of the alpha resourceClaims field introduced to Pods in 1.26.0 was modified from “set” to “map”, resolving an incompatibility with use of this schema in CustomResourceDefinitions and with server-side apply. (kubernetes/kubernetes#114617, @JoelSpeed) [SIG API Machinery]
‘A new preEnqueue extension point was added to scheduler’s component config
v1beta2/v1beta3/v1.’
(kubernetes/kubernetes#113275, @Huang-Wei)
‘Added a ResourceClaim API (in the resource.k8s.io/v1alpha1 API group and
behind the DynamicResourceAllocation feature gate).
The new API is now more flexible than the existing Device Plugins feature of Kubernetes because it
allows Pods to request (claim) special kinds of resources, which can be available at node level, cluster
level, or following any other model you implement.’ (kubernetes/kubernetes#111023, @pohly)
‘Container preStop and postStart lifecycle handlers using httpGet now
honor the specified scheme and headers fields. This enables setting custom
headers and changing the scheme to HTTPS, consistent with container
startup/readiness/liveness probe capabilities. Lifecycle handlers configured
with scheme: HTTPS that encounter errors indicating the endpoint is actually
using HTTP fall back to making the request over HTTP for compatibility with
previous releases. When this happens, a LifecycleHTTPFallback event is recorded
in the namespace of the pod and a kubelet_lifecycle_handler_http_fallbacks_total
metric in the kubelet is incremented. Cluster administrators can opt out of the
expanded lifecycle handler capabilities by setting
--feature-gates=ConsistentHTTPGetHandlers=false in kubelet.’
(kubernetes/kubernetes#86139, @jasimmons)
‘Graduated JobTrackingWithFinalizers to stable.
Jobs created before the feature was enabled are still tracked without finalizers.
Jobs tracked with finalizers have the annotation batch.kubernetes.io/job-tracking.
If the annotation is present and the user attempts to remove it, the control plane adds it back.
The annotation batch.kubernetes.io/job-tracking is now deprecated.
The control plane will ignore it and stop adding it for new Jobs in v1.27.’ (kubernetes/kubernetes#113510, @alculquicondor)
‘Kubelet added the following Pod failure conditions:
‘Priority and Fairness has introduced a new feature called borrowing that allows an API priority level
to borrow a number of seats from other priority level(s). As a cluster operator, you can enable borrowing
for a certain priority level configuration object via the two newly introduced fields lendablePercent, and
borrowingLimitPercent located under the .spec.limited field of the designated priority level.
This change added the following metrics:
apiserver_flowcontrol_nominal_limit_seats: Nominal number of execution seats configured for each priority levelapiserver_flowcontrol_lower_limit_seats: Configured lower bound on number of execution seats available to each priority levelapiserver_flowcontrol_upper_limit_seats: Configured upper bound on number of execution seats available to each priority levelapiserver_flowcontrol_demand_seats: Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level)apiserver_flowcontrol_demand_seats_high_watermark: High watermark, over last adjustment period, of demand_seatsapiserver_flowcontrol_demand_seats_average: Time-weighted average, over last adjustment period, of demand_seatsapiserver_flowcontrol_demand_seats_stdev: Time-weighted standard deviation, over last adjustment period, of demand_seatsapiserver_flowcontrol_demand_seats_smoothed: Smoothed seat demandsapiserver_flowcontrol_target_seats: Seat allocation targetsapiserver_flowcontrol_seat_fair_frac: Fair fraction of server’s concurrency to allocate to each priority level that can use itapiserver_flowcontrol_current_limit_seats: current derived number of execution seats available to each priority level
The possibility of borrowing means that the old metric apiserver_flowcontrol_request_concurrency_limit can no longer mean both the configured concurrency limit and the enforced concurrency limit. Henceforth it means the configured concurrency limit.’ (kubernetes/kubernetes#113485, @MikeSpreitzer)
‘NodeInclusionPolicy in podTopologySpread plugin is now enabled by default.’
(kubernetes/kubernetes#113500, @kerthcet)
‘PodDisruptionBudget now adds an alpha spec.unhealthyPodEvictionPolicy field.
When the PDBUnhealthyPodEvictionPolicy feature-gate is enabled in kube-apiserver,
setting this field to "AlwaysAllow" allows pods to be evicted if they do not
have a ready condition, regardless of whether the PodDisruptionBudget is currently
healthy.’
(kubernetes/kubernetes#113375, @atiratree)
‘metav1.LabelSelectors specified in API objects are now validated to ensure
they do not contain invalid label values that will error at time of use. Existing
invalid objects can be updated, but new objects are required to contain valid
label selectors.’
(kubernetes/kubernetes#113699, @liggitt)
Add percentageOfNodesToScore as a scheduler profile level parameter to API version v1. When a profile percentageOfNodesToScore is set, it will override global percentageOfNodesToScore. (kubernetes/kubernetes#112521, @yuanchen8911)
Add auth API to get self subject attributes (new selfsubjectreviews API is added).
The corresponding command for kubctl is provided - kubectl auth whoami. (kubernetes/kubernetes#111333, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]
Added kubernetes_feature_enabled metric series to track whether each active feature gate is enabled. (kubernetes/kubernetes#112690, @logicalhan)
Added a --topology-manager-policy-options flag to the kubelet to support fine tuning the topology manager policies. The first policy option, prefer-closest-numa-nodes, allows these policies to favor sets of NUMA nodes with shorter distance between nodes when making admission decisions. (kubernetes/kubernetes#112914, @PiotrProkop)
Added a feature that allows a StatefulSet to start numbering replicas from an arbitrary non-negative ordinal, using the .spec.ordinals.start field. (kubernetes/kubernetes#112744, @pwschuurman)
Added a kube-proxy flag (--iptables-localhost-nodeports, default true) to allow disabling NodePort services on loopback addresses. Note: this only applies to iptables mode and ipv4. (kubernetes/kubernetes#108250, @cyclinder)
Added a new namespace alpha field to DataSourceRef field in PersistentVolumeClaim API. (kubernetes/kubernetes#113186, @ttakahashi21)
Aggregated discovery will be alpha and can be toggled with the AggregatedDiscoveryEndpoint feature flag. (kubernetes/kubernetes#113171, @Jefftree)
Clarified the CFS quota as 100ms in the code comments and set the minimum cpuCFSQuotaPeriod to 1ms to match Linux kernel expectations. (kubernetes/kubernetes#112123, @paskal)
Component-base: make the validation logic about LeaderElectionConfiguration consistent between component-base and client-go (kubernetes/kubernetes#111758, @SataQiu) [SIG API Machinery and Scheduling]
Deprecated the apiserver_request_slo_duration_seconds metric for v1.27 in favor of apiserver_request_sli_duration_seconds for naming consistency purposes with other SLI-specific metrics and to avoid any confusion between SLOs and SLIs. (kubernetes/kubernetes#112679, @dgrisonnet)
Enable the “Retriable and non-retriable pod failures for jobs” feature into beta. (kubernetes/kubernetes#113360, @mimowo)
Enabled kube-controller-manager to support ‘–concurrent-horizontal-pod-autoscaler-syncs’ flag to set the number of horizontal pod autoscaler controller workers. (kubernetes/kubernetes#108501, @zroubalik)
Fixed spurious field is immutable errors validating updates to Event API objects via the events.k8s.io/v1 API. (kubernetes/kubernetes#112183, @liggitt)
Graduated ServiceInternalTrafficPolicy feature to GA. (kubernetes/kubernetes#113496, @avoltz)
In ‘kube-proxy`: The “userspace” proxy mode (deprecated for over a year) is no
longer supported on either Linux or Windows. Users should use “iptables” or “ipvs”
on Linux, or “kernelspace” on Windows.
(kubernetes/kubernetes#112133, @knabben)
Introduce v1beta3 for Priority and Fairness with the following changes to the API spec:
- rename ‘assuredConcurrencyShares’ (located under `spec.limited’) to ‘nominalConcurrencyShares’.
- apply strategic merge patch annotations to ‘Conditions’ of flowschemas and
prioritylevelconfigurations. (kubernetes/kubernetes#112306, @tkashem)
Introduced v1alpha1 API for validating admission policies, enabling extensible admission control via CEL expressions (KEP 3488: CEL for Admission Control). To use, enable the ValidatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API via --runtime-config. (kubernetes/kubernetes#113314, @cici37)
KMS: added validation for duplicate kms config name when auto reload is enabled. If you enabled automatic reload of encryption configuration with API server flag --encryption-provider-config-automatic-reload, ensure all the KMS provider names (v1 and v2) in the encryption configuration are unique. (kubernetes/kubernetes#113697, @aramase)
Kubelet external Credential Provider feature is moved to GA. Credential Provider Plugin and Credential Provider Config APIs updated from v1beta1 to v1 with no API changes. (kubernetes/kubernetes#111616, @ndixita)
Legacy klog flags are no longer available. Only -v and -vmodule are still supported. (kubernetes/kubernetes#112120, @pohly) [SIG Architecture, CLI, Instrumentation, Node and Testing]
Moved MixedProtocolLBService from beta to GA. (kubernetes/kubernetes#112895, @janosi)
New Pod API field .spec.schedulingGates is introduced to enable users to control when to mark a Pod as scheduling ready. (kubernetes/kubernetes#113274, @Huang-Wei)
Protobuf serialization of metav1.MicroTime timestamps (used in Lease and Event API objects) has been corrected to truncate to microsecond precision, to match the documented behavior and JSON/YAML serialization. Any existing persisted data is truncated to microsecond when read from etcd. (kubernetes/kubernetes#111936, @haoruan)
Removed feature gates ServiceLoadBalancerClass and ServiceLBNodePortControl. These feature gates were enabled (and locked) since v1.24. (kubernetes/kubernetes#112577, @andrewsykim)
Reverted regression that prevented client-go latency metrics to be reported with a template URL to avoid label cardinality. (kubernetes/kubernetes#111752, @aanm)
The EndpointSliceTerminatingCondition feature gate was graduated to GA. The gate is now locked and will be removed in v1.28. (kubernetes/kubernetes#113351, @andrewsykim)
DynamicKubeletConfig feature gate has been removed from the API server.
Dynamic kubelet reconfiguration now can’t be used even when older nodes are still
attempting to rely on it. This is aligned with the Kubernetes version skew policy.
(kubernetes/kubernetes#112643, @SergeyKanzhelev)
kubectl wait command with jsonpath flag will wait for target path until timeout.
(kubernetes/kubernetes#109525, @jonyhy96)
Add a ResourceClaim API (in the resource.k8s.io/v1alpha1 API group and
behind the DynamicResourceAllocation feature gate).
The new API is more flexible than the existing Device Plugins feature of Kubernetes because it
allows Pods to request (claim) special kinds of resources, which can be available at node level, cluster
level, or following any other model you implement. (kubernetes/kubernetes#111023, @pohly) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Release, Scheduling, Storage and Testing]
PodDisruptionBudget adds an alpha spec.unhealthyPodEvictionPolicy field. When the PDBUnhealthyPodEvictionPolicy feature-gate is enabled in kube-apiserver, setting this field to "AlwaysAllow" allows pods to be evicted if they do not have a ready condition, regardless of whether the PodDisruptionBudget is currently healthy. (kubernetes/kubernetes#113375, @atiratree) [SIG API Machinery, Apps, Auth and Testing]
A new preEnqueue extension point is added to scheduler’s component config v1beta2/v1beta3/v1. (kubernetes/kubernetes#113275, @Huang-Wei) [SIG API Machinery, Apps, Instrumentation, Scheduling and Testing]
Add a new namespace alpha field to dataSourceRef field in PersistentVolumeClaim API. (kubernetes/kubernetes#113186, @ttakahashi21) [SIG API Machinery, Apps, Storage and Testing]
Add a kube-proxy flag (–iptables-localhost-nodeports, default true) to allow disabling NodePort services on loopback addresses. Note: this only applies to iptables mode and ipv4. (kubernetes/kubernetes#108250, @cyclinder) [SIG API Machinery, Cloud Provider, Network, Node, Scalability, Storage and Testing]
Added a –topology-manager-policy-options flag to the kubelet to support fine tuning the topology manager policies. The first policy option, prefer-closest-numa-nodes, allows these policies to favor sets of NUMA nodes with shorter distance between nodes when making admission decisions. (kubernetes/kubernetes#112914, @PiotrProkop) [SIG API Machinery and Node]
Added a feature that allows a StatefulSet to start numbering replicas from an arbitrary non-negative ordinal, using the .spec.ordinals.start field. (kubernetes/kubernetes#112744, @pwschuurman) [SIG API Machinery and Apps]
Deprecate the apiserver_request_slo_duration_seconds metric for v1.27 in favor of apiserver_request_sli_duration_seconds for naming consistency purposes with other SLI-specific metrics and to avoid any confusion between SLOs and SLIs. (kubernetes/kubernetes#112679, @dgrisonnet) [SIG API Machinery and Instrumentation]
Enable the “Retriable and non-retriable pod failures for jobs” feature into beta (kubernetes/kubernetes#113360, @mimowo) [SIG Apps, Auth, Node, Scheduling and Testing]
Graduate JobTrackingWithFinalizers to stable.
Jobs created before the feature was enabled are still tracked without finalizers.
Users can choose to migrate jobs to tracking with finalizers by adding the annotation batch.kubernetes.io/job-tracking.
If the annotation was already present and the user attempts to remove it, the control plane adds the annotation back. (kubernetes/kubernetes#113510, @alculquicondor) [SIG API Machinery, Apps and Testing]
Graduate ServiceInternalTrafficPolicy feature to GA (kubernetes/kubernetes#113496, @avoltz) [SIG Apps and Network]
If you enabled automatic reload of encryption configuration with API server flag –encryption-provider-config-automatic-reload, ensure all the KMS provider names (v1 and v2) in the encryption configuration are unique. (kubernetes/kubernetes#113697, @aramase) [SIG API Machinery and Auth]
Introduce v1alpha1 API for validating admission policies, enabling extensible admission control via CEL expressions (KEP 3488: CEL for Admission Control). To use, enable the ValidatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API via --runtime-config. (kubernetes/kubernetes#113314, @cici37) [SIG API Machinery, Auth, Cloud Provider and Testing]
Kubelet adds the following pod failure conditions:
Metav1.LabelSelectors specified in API objects are now validated to ensure they do not contain invalid label values that will error at time of use. Existing invalid objects can be updated, but new objects are required to contain valid label selectors. (kubernetes/kubernetes#113699, @liggitt) [SIG API Machinery, Apps, Auth, Network and Storage]
Moving MixedProtocolLBService from beta to GA (kubernetes/kubernetes#112895, @janosi) [SIG Apps, Network and Testing]
New Pod API field .spec.schedulingGates is introduced to enable users to control when to mark a Pod as scheduling ready. (kubernetes/kubernetes#113274, @Huang-Wei) [SIG Apps, Scheduling and Testing]
NodeInclusionPolicy in podTopologySpread plugin is enabled by default. (kubernetes/kubernetes#113500, @kerthcet) [SIG API Machinery, Apps, Scheduling and Testing]
Priority and Fairness has introduced a new feature called borrowing that allows an API priority level
to borrow a number of seats from other priority level(s). As a cluster operator, you can enable borrowing
for a certain priority level configuration object via the two newly introduced fields lendablePercent, and
borrowingLimitPercent located under the .spec.limited field of the designated priority level.
This PR adds the following metrics.
apiserver_flowcontrol_nominal_limit_seats: Nominal number of execution seats configured for each priority levelapiserver_flowcontrol_lower_limit_seats: Configured lower bound on number of execution seats available to each priority levelapiserver_flowcontrol_upper_limit_seats: Configured upper bound on number of execution seats available to each priority levelapiserver_flowcontrol_demand_seats: Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level)apiserver_flowcontrol_demand_seats_high_watermark: High watermark, over last adjustment period, of demand_seatsapiserver_flowcontrol_demand_seats_average: Time-weighted average, over last adjustment period, of demand_seatsapiserver_flowcontrol_demand_seats_stdev: Time-weighted standard deviation, over last adjustment period, of demand_seatsapiserver_flowcontrol_demand_seats_smoothed: Smoothed seat demandsapiserver_flowcontrol_target_seats: Seat allocation targetsapiserver_flowcontrol_seat_fair_frac: Fair fraction of server’s concurrency to allocate to each priority level that can use itapiserver_flowcontrol_current_limit_seats: current derived number of execution seats available to each priority level
The possibility of borrowing means that the old metric apiserver_flowcontrol_request_concurrency_limit can no longer mean both the configured concurrency limit and the enforced concurrency limit. Henceforth it means the configured concurrency limit. (kubernetes/kubernetes#113485, @MikeSpreitzer) [SIG API Machinery and Testing]
The EndpointSliceTerminatingCondition feature gate has graduated to GA. The gate is now locked and will be removed in v1.28. (kubernetes/kubernetes#113351, @andrewsykim) [SIG API Machinery, Apps, Network and Testing]
Yes, aggregated discovery will be alpha and can be toggled with the AggregatedDiscoveryEndpoint feature flag (kubernetes/kubernetes#113171, @Jefftree) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Network, Node, Release, Scalability, Scheduling, Storage and Testing]
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
(kubernetes/kubernetes#86139, @jasimmons) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Contributor Experience, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]Add percentageOfNodesToScore as a scheduler profile level parameter to API version v1. If a profile percentageOfNodesToScore is set, it will override global percentageOfNodesToScore. (kubernetes/kubernetes#112521, @yuanchen8911) [SIG API Machinery, Scheduling and Testing]
Kube-controller-manager supports ‘–concurrent-horizontal-pod-autoscaler-syncs’ flag to set the number of horizontal pod autoscaler controller workers. (kubernetes/kubernetes#108501, @zroubalik) [SIG API Machinery, Apps and Autoscaling]
Kube-proxy: The “userspace” proxy mode (deprecated for over a year) is no longer supported on either Linux or Windows. Users should use “iptables” or “ipvs” on Linux, or “kernelspace” on Windows. (kubernetes/kubernetes#112133, @knabben) [SIG API Machinery, Network, Scalability, Testing and Windows]
Kubectl wait command with jsonpath flag will wait for target path appear until timeout. (kubernetes/kubernetes#109525, @jonyhy96) [SIG CLI and Testing]
Kubelet external Credential Provider feature is moved to GA. Credential Provider Plugin and Credential Provider Config APIs updated from v1beta1 to v1 with no API changes. (kubernetes/kubernetes#111616, @ndixita) [SIG API Machinery, Node, Scheduling and Testing]
The DynamicKubeletConfig feature gate has been removed from the API server. Dynamic kubelet reconfiguration now cannot be used even when older nodes are still attempting to rely on it. This is aligned with the Kubernetes version skew policy. (kubernetes/kubernetes#112643, @SergeyKanzhelev) [SIG API Machinery, Apps, Auth, Node and Testing]
Add kubernetes_feature_enabled metric series to track whether each active feature gate is enabled. (kubernetes/kubernetes#112690, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle, Instrumentation, Network, Node and Scheduling]
Introduce v1beta3 for Priority and Fairness with the following changes to the API spec:
- rename ‘assuredConcurrencyShares’ (located under spec.limited’) to ‘nominalConcurrencyShares’
- apply strategic merge patch annotations to ‘Conditions’ of flowschemas and prioritylevelconfigurations (kubernetes/kubernetes#112306, @tkashem) [SIG API Machinery and Testing]
Legacy klog flags are no longer available. Only -v and -vmodule are still supported. (kubernetes/kubernetes#112120, @pohly) [SIG Architecture, CLI, Instrumentation, Node and Testing]
The feature gates ServiceLoadBalancerClass and ServiceLBNodePortControl have been removed. These feature gates were enabled (and locked) since v1.24. (kubernetes/kubernetes#112577, @andrewsykim) [SIG Apps]
Add auth API to get self subject attributes (new selfsubjectreviews API is added).
The corresponding command for kubctl is provided - kubectl auth whoami. (kubernetes/kubernetes#111333, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]
Clarified the CFS quota as 100ms in the code comments and set the minimum cpuCFSQuotaPeriod to 1ms to match Linux kernel expectations. (kubernetes/kubernetes#112123, @paskal) [SIG API Machinery and Node]
Component-base: make the validation logic about LeaderElectionConfiguration consistent between component-base and client-go (kubernetes/kubernetes#111758, @SataQiu) [SIG API Machinery and Scheduling]
Fixes spurious field is immutable errors validating updates to Event API objects via the events.k8s.io/v1 API (kubernetes/kubernetes#112183, @liggitt) [SIG Apps]
Protobuf serialization of metav1.MicroTime timestamps (used in Lease and Event API objects) has been corrected to truncate to microsecond precision, to match the documented behavior and JSON/YAML serialization. Any existing persisted data is truncated to microsecond when read from etcd. (kubernetes/kubernetes#111936, @haoruan) [SIG API Machinery]
Revert regression that prevented client-go latency metrics to be reported with a template URL to avoid label cardinality. (kubernetes/kubernetes#111752, @aanm) [SIG API Machinery]
[kubelet] Change default cpuCFSQuotaPeriod value with enabled cpuCFSQuotaPeriod flag from 100ms to 100µs to match the Linux CFS and k8s defaults. cpuCFSQuotaPeriod of 100ms now requires customCPUCFSQuotaPeriod flag to be set to work. (kubernetes/kubernetes#111520, @paskal) [SIG API Machinery and Node]
Kube-apiserver: Fixes handling of CRD schemas containing literal null values in enums (#104988, @liggitt) [SIG API Machinery, Apps and Network]
A new score extension for NodeResourcesFit plugin that merges the functionality of NodeResourcesLeastAllocated, NodeResourcesMostAllocated, RequestedToCapacityRatio plugins, which are marked as deprecated as of v1beta2. In v1beta1, the three plugins can still be used in v1beta1 but not at the same time with the score extension of NodeResourcesFit. (kubernetes/kubernetes#101822, @yuzhiquan)
A value of Auto is now a valid for the service.kubernetes.io/topology-aware-hints annotation. (kubernetes/kubernetes#100728, @robscott)
Add DataSourceRef alpha field to PVC spec, which allows contents other than PVCs and VolumeSnapshots to be data sources. (kubernetes/kubernetes#103276, @bswartz)
Add PersistentVolumeClaimDeletePoilcy to StatefulSet API. (kubernetes/kubernetes#99378, @mattcary)
Add a new Priority and Fairness rule that exempts all probes (/readyz, /healthz, /livez) to prevent restarting of healthy kube-apiserver instance by kubelet. (kubernetes/kubernetes#100678, @tkashem)
Add alpha support for HostProcess containers on Windows (kubernetes/kubernetes#99576, @marosset) [SIG API Machinery, Apps, Node, Testing and Windows]
Add distributed tracing to the kube-apiserver. It is can be enabled with the feature gate APIServerTracing (kubernetes/kubernetes#94942, @dashpole)
Add three metrics to the job controller to monitor if a job works in healthy condition.
IndexedJob has been promoted to Beta. (kubernetes/kubernetes#101292, @AliceZhang2016)
Added field .status.uncountedTerminatedPods to the Job resource. This field is used by the job controller to keep track of finished pods before adding them to the Job status counters. Pods created by the job controller get the finalizer batch.kubernetes.io/job-tracking
Jobs that are tracked using this mechanism get the annotation batch.kubernetes.io/job-tracking. This is a temporary measure. Two releases after this feature graduates to beta, the annotation won’t be added to Jobs anymore. (kubernetes/kubernetes#98817, @alculquicondor)
Added new kubelet alpha feature SeccompDefault. This feature enables falling back to
the RuntimeDefault (former runtime/default) seccomp profile if nothing else is specified
in the pod/container SecurityContext or the pod annotation level. To use the feature, enable
the feature gate as well as set the kubelet configuration option SeccompDefault
(--seccomp-default) to true. (kubernetes/kubernetes#101943, @saschagrunert) [SIG Node]
Adds the ReadWriteOncePod access mode for PersistentVolumes and PersistentVolumeClaims. Restricts volume access to a single pod on a single node. (kubernetes/kubernetes#102028, @chrishenzie)
Alpha swap support can now be enabled on Kubernetes nodes with the NodeSwapEnabled feature flag. See KEP-2400 for details. (kubernetes/kubernetes#102823, @ehashman)
Because of the implementation logic of time.Format in golang, the displayed time zone is not consistent. (kubernetes/kubernetes#102366, @cndoit18)
Corrected the documentation for escaping dollar signs in a container’s env, command and args property. (kubernetes/kubernetes#101916, @MartinKanters) [SIG Apps]
Enable MaxSurge for DaemonSet by default. (kubernetes/kubernetes#101742, @ravisantoshgudimetla)
Enforce the ReadWriteOncePod PVC access mode during scheduling (kubernetes/kubernetes#103082, @chrishenzie)
Ephemeral containers are now allowed to configure a securityContext that differs from that of the Pod. Cluster administrators should ensure that security policy controllers support EphemeralContainers before enabling this feature in clusters. (kubernetes/kubernetes#99023, @verb)
Exec plugin authors can override default handling of standard input via new interactiveMode kubeconfig field. (kubernetes/kubernetes#99310, @ankeesler)
If someone had the ProbeTerminationGracePeriod alpha feature enabled in 1.21, they should update/delete any workloads/pods with probe terminationGracePeriods < 1 before upgrading (kubernetes/kubernetes#103245, @wzshiming)
Improved parsing of label selectors (kubernetes/kubernetes#102188, @alculquicondor) [SIG API Machinery]
Introduce minReadySeconds api to the StatefulSets. (kubernetes/kubernetes#100842, @ravisantoshgudimetla)
Introducing Memory quality of service support with cgroups v2 (Alpha). The MemoryQoS feature is now in Alpha. This allows kubelet running with cgroups v2 to set memory QoS at container, pod and QoS level to protect and guarantee better memory quality. This feature can be enabled through feature gate Memory QoS. (kubernetes/kubernetes#102970, @borgerli)
Kube API server accepts Impersonate-Uid header to impersonate a user with a specific UID, in the same way that you can currently use Impersonate-User, Impersonate-Group and Impersonate-Extra. (kubernetes/kubernetes#99961, @margocrawf)
Kube-apiserver: --service-account-issuer can be specified multiple times now, to enable non-disruptive change of issuer. (kubernetes/kubernetes#101155, @zshihang) [SIG API Machinery, Auth, Node and Testing]
Kube-controller-manager: the --horizontal-pod-autoscaler-use-rest-clients flag and Heapster support in the horizontal pod autoscaler, deprecated since 1.12, is removed. (kubernetes/kubernetes#90368, @serathius)
Kube-scheduler: a plugin enabled in a v1beta2 configuration file takes precedence over the default configuration for that plugin. This simplifies enabling default plugins with custom configuration without needing to explicitly disable those default plugins. (kubernetes/kubernetes#99582, @chendave)
New node-high priority-level has been added to Suggested API Priority and (kubernetes/kubernetes#101151, @mborsz)
NodeSwapEnabled feature flag was renamed to NodeSwap
The flag was only available in the 1.22.0-beta.1 release, and the new flag should be used going forward. (kubernetes/kubernetes#103553, @ehashman) [SIG Node]
Omit comparison with boolean constant (kubernetes/kubernetes#101523, @chuntaochen) [SIG CLI and Cloud Provider]
Removed the feature flag for probe-level termination grace period from Kubelet. If a user wants to disable this feature on already created pods, they will have to delete and recreate the pods. (kubernetes/kubernetes#103168, @raisaat) [SIG Apps and Node]
Revert addition of Add PersistentVolumeClaimDeletePoilcy to StatefulSetAPI. (kubernetes/kubernetes#103747, @mattcary)
Scheduler could be configured to consider new resources beside CPU and memory, GPU for example, for the score plugin of NodeResourcesBalancedAllocation. (kubernetes/kubernetes#101946, @chendave) [SIG Scheduling]
Server Side Apply now treats all Selector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. (kubernetes/kubernetes#97989, @Danil-Grigorev) [SIG API Machinery]
Suspend Job feature graduated to beta. Added the action label to Job controller sync metrics job_sync_total and job_sync_duration_seconds. (kubernetes/kubernetes#102022, @adtac)
The API documentation for the DaemonSet’s spec.updateStrategy.rollingUpdate.maxUnavailable field was corrected to state that the value is rounded up. (kubernetes/kubernetes#101296, @Miciah)
The CSIServiceAccountToken graduates to Ga and is unconditionally enabled. (kubernetes/kubernetes#103001, @zshihang)
The CertificateSigningRequest.certificates.k8s.io API supports an optional expirationSeconds field to allow the client to request a particular duration for the issued certificate. The default signer implementations provided by the Kubernetes controller manager will honor this field as long as it does not exceed the –cluster-signing-duration flag. (kubernetes/kubernetes#99494, @enj)
The EndpointSlicen Mirroring controller no longer mirrors the last-applied-configuration annotation created by kubectl to update EndpointSlices. (kubernetes/kubernetes#102731, @sharmarajdaksh)
The NetworkPolicyEndPort is graduated to beta and is enabled by default. (kubernetes/kubernetes#102834, @rikatz)
The PodDeletionCost feature has been promoted to beta, and enabled by default. (kubernetes/kubernetes#101080, @ahg-g)
The Server Side Apply treats certain structs as atomic. Meaning the entire selector field is managed by a single writer and updated together. (kubernetes/kubernetes#100684, @Jefftree)
The ServiceAppProtocol feature gate has been removed. It reached GA in Kubernetes (kubernetes/kubernetes#103190, @robscott)
The TerminationGracePeriodSeconds on pod specs and container probes should not be negative. Negative values of TerminationGracePeriodSeconds will be treated as the value 1s on the delete path. Immutable field validation will be relaxed in order to update negative values. In a future release, negative values will not be permitted. (kubernetes/kubernetes#98866, @wzshiming)
The kube-scheduler component config v1beta2 API available
Three scheduler plugins deprecated (NodeLabel, ServiceAffinity, NodePreferAvoidPods). (kubernetes/kubernetes#99597, @adtac)
The pod/eviction subresource now accepts policy/v1 eviction requests in addition to policy/v1beta1 eviction requests (kubernetes/kubernetes#100724, @liggitt)
The podAffinity, NamespaceSelector and the associated CrossNamespaceAffinity quota scope features graduate to Beta and they are now enabled by default. (kubernetes/kubernetes#101496, @ahg-g)
The pods/ephemeralcontainers API now returns and expects a Pod object instead of EphemeralContainers. This is incompatible with the previous alpha-level API. (kubernetes/kubernetes#101034, @verb) [SIG Apps, Auth, CLI and Testing]
The v1.Node and .status.images[].names are now optional. (kubernetes/kubernetes#102159, @roycaihw)
The deprecated flag --algorithm-provider has been removed from kube-scheduler. Use instead ComponentConfig to configure the set of enabled plugins. (kubernetes/kubernetes#102239, @Haleygo)
The options --ssh-user and --ssh-key are removed. They only functioned on GCE, and only in-tree. Use the apiserver network proxy instead. (kubernetes/kubernetes#102297, @deads2k)
Track Job completion through status and Pod finalizers, removing dependency on Pod tombstones. (kubernetes/kubernetes#98238, @alculquicondor) [SIG API Machinery, Apps, Auth and Testing]
Track ownership of scale subresource for all scalable resources i.e. Deployment, ReplicaSet, StatefulSet, ReplicationController, and Custom Resources. (kubernetes/kubernetes#98377, @nodo) [SIG API Machinery and Testing]
Revert addition of Add PersistentVolumeClaimDeletePoilcy to StatefulSet API. (kubernetes/kubernetes#103747, @mattcary) [SIG API Machinery and Apps]
Added field .status.uncountedTerminatedPods to the Job resource. This field is used by the job controller to keep track of finished pods before adding them to the Job status counters.
Pods created by the job controller get the finalizer batch.kubernetes.io/job-tracking
Jobs that are tracked using this mechanism get the annotation batch.kubernetes.io/job-tracking. This is a temporary measure. Two releases after this feature graduates to beta, the annotation won’t be added to Jobs anymore. (kubernetes/kubernetes#98817, @alculquicondor) [SIG API Machinery, Apps, Auth and CLI]
Ephemeral containers are now allowed to configure a securityContext that differs from that of the Pod.
Cluster administrators should ensure that security policy controllers support EphemeralContainers before enabling this feature in clusters. (kubernetes/kubernetes#99023, @verb) [SIG API Machinery, Apps, Auth and Node]
If someone had the ProbeTerminationGracePeriod alpha feature enabled in 1.21, they should update/delete any workloads/pods with probe terminationGracePeriods < 1 before upgrading (kubernetes/kubernetes#103245, @wzshiming) [SIG Apps and Node]
Introducing Memory QoS support with cgroups v2 (Alpha)
The MemoryQoS feature is now in Alpha. This allows kubelet running with cgroups v2 to set memory QoS at container, pod and QoS level to protect and guarantee better memory quality. This feature can be enabled through feature gate MemoryQoS. (kubernetes/kubernetes#102970, @borgerli) [SIG Node and Storage]
NodeSwapEnabled feature flag was renamed to NodeSwap
The flag was only available in the 1.22.0-beta.1 release, and the new flag should be used going forward. (kubernetes/kubernetes#103553, @ehashman) [SIG Node]
Removed the feature flag for probe-level termination grace period from Kubelet. If a user wants to disable this feature on already created pods, they will have to delete and recreate the pods. (kubernetes/kubernetes#103168, @raisaat) [SIG Apps and Node]
Track Job completion through status and Pod finalizers, removing dependency on Pod tombstones. (kubernetes/kubernetes#98238, @alculquicondor) [SIG API Machinery, Apps, Auth and Testing]
When using kubectl replace (or the equivalent API call) on a Service, the caller no longer needs to do a read-modify-write cycle to fetch the allocated values for .spec.clusterIP and .spec.ports[].nodePort. Instead the API server will automatically carry these forward from the original object when the new object does not specify them. (kubernetes/kubernetes#103532, @thockin) [SIG Apps and Network]
A new score extension for NodeResourcesFit plugin that merges the functionality of NodeResourcesLeastAllocated,NodeResourcesMostAllocated,RequestedToCapacityRatio plugins, which are marked as deprecated as of v1beta2. In v1beta1, the three plugins can still be used in v1beta1 but not at the same time with the score extension of NodeResourcesFit
Add DataSourceRef alpha field to PVC spec, which allows contents other than PVCs and VolumeSnapshots to be data sources. (kubernetes/kubernetes#103276, @bswartz) [SIG API Machinery, Apps and Storage]
Add PersistentVolumeClaimDeletePoilcy to StatefulSet API. (kubernetes/kubernetes#99378, @mattcary) [SIG API Machinery and Apps]
Add distributed tracing to the kube-apiserver. It is can be enabled with the feature gate: APIServerTracing=true (kubernetes/kubernetes#94942, @dashpole) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]
Added new kubelet alpha feature SeccompDefault. This feature enables falling back to
the RuntimeDefault (former runtime/default) seccomp profile if nothing else is specified
in the pod/container SecurityContext or the pod annotation level. To use the feature, enable
the feature gate as well as set the kubelet configuration option SeccompDefault
(--seccomp-default) to true. (kubernetes/kubernetes#101943, @saschagrunert) [SIG Node]
Adds the ReadWriteOncePod access mode for PersistentVolumes and PersistentVolumeClaims. Restricts volume access to a single pod on a single node. (kubernetes/kubernetes#102028, @chrishenzie) [SIG Apps, CLI, Node, Scheduling and Storage]
Alpha swap support can now be enabled on Kubernetes nodes with the NodeSwapEnabled feature flag. See for details. (kubernetes/kubernetes#102823, @ehashman) [SIG Node]
CSIServiceAccountToken is GA. (kubernetes/kubernetes#103001, @zshihang) [SIG Auth and Storage]
Enforce the ReadWriteOncePod PVC access mode during scheduling (kubernetes/kubernetes#103082, @chrishenzie) [SIG Apps, CLI, Node, Scheduling and Storage]
Improved parsing of label selectors (kubernetes/kubernetes#102188, @alculquicondor) [SIG API Machinery]
Kube API server accepts Impersonate-Uid header to impersonate a user with a specific UID, in the same way that you can currently use Impersonate-User, Impersonate-Group and Impersonate-Extra (kubernetes/kubernetes#99961, @margocrawf) [SIG API Machinery, Auth and Testing]
Kube-scheduler: a plugin enabled in a v1beta2 configuration file takes precedence over the default configuration for that plugin; this simplifies enabling default plugins with custom configuration without needing to explicitly disable those default plugins. (kubernetes/kubernetes#99582, @chendave) [SIG Scheduling]
Scheduler could be configured to consider new resources beside CPU and memory, GPU for example, for the score plugin of NodeResourcesBalancedAllocation. (kubernetes/kubernetes#101946, @chendave) [SIG Scheduling]
Suspend Job feature graduated to beta
Added the “action” label to Job controller sync metrics job_sync_total and job_sync_duration_seconds (kubernetes/kubernetes#102022, @adtac) [SIG Apps, Instrumentation and Testing]
TerminationGracePeriodSeconds on pod specs and container probes should not be negative.
Negative values of TerminationGracePeriodSeconds will be treated as the value 1s on the delete path.
Immutable field validation will be relaxed in order to update negative values.
In a future release, negative values will not be permitted. (kubernetes/kubernetes#98866, @wzshiming) [SIG API Machinery, Apps and Node]
The API documentation for the DaemonSet’s spec.updateStrategy.rollingUpdate.maxUnavailable field was corrected to state that the value is rounded up. (kubernetes/kubernetes#101296, @Miciah) [SIG Apps and CLI]
The CertificateSigningRequest.certificates.k8s.io API supports an optional expirationSeconds field to allow the client to request a particular duration for the issued certificate. The default signer implementations provided by the Kubernetes controller manager will honor this field as long as it does not exceed the –cluster-signing-duration flag. (kubernetes/kubernetes#99494, @enj) [SIG API Machinery, Apps, Auth, CLI, Instrumentation, Node, Security and Testing]
The ServiceAppProtocol feature gate has been removed. It reached GA in Kubernetes 1.20. (kubernetes/kubernetes#103190, @robscott) [SIG Network]
Because of the implementation logic of time.Format in golang, the displayed time zone is not consistent (kubernetes/kubernetes#102366, @cndoit18) [SIG Apps, Auth, Autoscaling, CLI, Cluster Lifecycle, Instrumentation, Network, Node and Testing]
Endpoint slices mirroring controller no longer mirrors the last-applied-configuration annotation created by kubectl to updated endpoint slices (kubernetes/kubernetes#102731, @sharmarajdaksh) [SIG API Machinery, Apps, Cloud Provider, Network, Release, Scheduling, Storage and Testing]
Exec plugin authors can override default handling of standard input via new interactiveMode kubeconfig field (kubernetes/kubernetes#99310, @ankeesler) [SIG API Machinery, Auth, CLI and Testing]
Kube-scheduler component config v1beta2 API available
Three scheduler plugins deprecated (NodeLabel, ServiceAffinity, NodePreferAvoidPods) (kubernetes/kubernetes#99597, @adtac) [SIG Scheduling]
Network Policy EndPort is graduated to beta and is enabled by default (kubernetes/kubernetes#102834, @rikatz) [SIG Network]
–ssh-user and –ssh-key options are removed. They only functioned on GCE, and only in-tree. Use the apiserver network proxy instead. (kubernetes/kubernetes#102297, @deads2k) [SIG API Machinery, Cloud Provider and Testing]
Enable MaxSurge for DS by default (kubernetes/kubernetes#101742, @ravisantoshgudimetla) [SIG Apps and Testing]
Introduce minReadySeconds api to the StatefulSets. (kubernetes/kubernetes#100842, @ravisantoshgudimetla) [SIG API Machinery, Apps and Testing]
Kube-controller-manger: the --horizontal-pod-autoscaler-use-rest-clients flag and Heapster support in the horizontal pod autoscaler, deprecated since 1.12, is removed. (kubernetes/kubernetes#90368, @serathius) [SIG API Machinery, Apps, Autoscaling, Cloud Provider and Instrumentation]
The deprecated flag –algorithm-provider has been removed from kube-scheduler. Use instead ComponentConfig to configure the set of enabled plugins (kubernetes/kubernetes#102239, @Haleygo) [SIG Cloud Provider and Scheduling]
Add alpha support for HostProcess containers on Windows (kubernetes/kubernetes#99576, @marosset) [SIG API Machinery, Apps, Node, Testing and Windows]
Add three metrics to job controller to monitor if Job works in a healthy condition.
IndexedJob promoted to Beta (kubernetes/kubernetes#101292, @AliceZhang2016) [SIG Apps, Instrumentation and Testing]
Corrected the documentation for escaping dollar signs in a container’s env, command and args property. (kubernetes/kubernetes#101916, @MartinKanters) [SIG Apps]
Omit comparison with boolean constant (kubernetes/kubernetes#101523, @GreenApple10) [SIG CLI and Cloud Provider]
Pod Affinity NamespaceSelector and the associated CrossNamespaceAffinity quota scope graduated to beta (kubernetes/kubernetes#101496, @ahg-g) [SIG API Machinery, Apps and Testing]
V1.Node .status.images[].names is now optional (kubernetes/kubernetes#102159, @roycaihw) [SIG Apps and Node]
“Auto” is now a valid value for the service.kubernetes.io/topology-aware-hints annotation. (kubernetes/kubernetes#100728, @robscott) [SIG Apps, Instrumentation and Network]
Kube-apiserver: --service-account-issuer can be specified multiple times now, to enable non-disruptive change of issuer. (kubernetes/kubernetes#101155, @zshihang) [SIG API Machinery, Auth, Node and Testing]
New “node-high” priority-level has been added to Suggested API Priority and Fairness configuration. (kubernetes/kubernetes#101151, @mborsz) [SIG API Machinery]
PodDeletionCost promoted to Beta (kubernetes/kubernetes#101080, @ahg-g) [SIG Apps]
SSA treats certain structs as atomic (kubernetes/kubernetes#100684, @Jefftree) [SIG API Machinery, Auth, Node and Storage]
Server Side Apply now treats all Selector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. (kubernetes/kubernetes#97989, @Danil-Grigorev) [SIG API Machinery]
The pods/ephemeralcontainers API now returns and expects a Pod object instead of EphemeralContainers. This is incompatible with the previous alpha-level API. (kubernetes/kubernetes#101034, @verb) [SIG Apps, Auth, CLI and Testing]
The pod/eviction subresource now accepts policy/v1 Eviction requests in addition to policy/v1beta1 Eviction requests (kubernetes/kubernetes#100724, @liggitt) [SIG API Machinery, Apps, Architecture, Auth, CLI, Storage and Testing]
Track ownership of scale subresource for all scalable resources i.e. Deployment, ReplicaSet, StatefulSet, ReplicationController, and Custom Resources. (kubernetes/kubernetes#98377, @nodo) [SIG API Machinery and Testing]
We have added a new Priority & Fairness rule that exempts all probes (/readyz, /healthz, /livez) to prevent
restarting of “healthy” kube-apiserver instance(s) by kubelet. (kubernetes/kubernetes#100678, @tkashem) [SIG API Machinery]
We have added a new Priority & Fairness rule that exempts all probes (/readyz, /healthz, /livez) to prevent
restarting of “healthy” kube-apiserver instance(s) by kubelet. (kubernetes/kubernetes#101112, @tkashem) [SIG API Machinery]
Fixes using server-side apply with APIService resources (kubernetes/kubernetes#100714, @kevindelgado) [SIG API Machinery, Apps and Testing]
Regenerate protobuf code to fix CVE-2021-3121 (kubernetes/kubernetes#100501, @joelsmith) [SIG API Machinery, Apps, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage]
Kubernetes is now built using go1.15.8 (kubernetes/kubernetes#98962, @cpanato) [SIG Cloud Provider, Instrumentation, Release and Testing]
TokenRequest and TokenRequestProjection features have been promoted to GA. This feature allows generating service account tokens that are not visible in Secret objects and are tied to the lifetime of a Pod object. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection for details on configuring and using this feature. The TokenRequest and TokenRequestProjection feature gates will be removed in v1.21.
- kubeadm’s kube-apiserver Pod manifest now includes the following flags by default “–service-account-key-file”, “–service-account-signing-key-file”, “–service-account-issuer”. (kubernetes/kubernetes#93258, @zshihang) [SIG API Machinery, Auth, Cluster Lifecycle, Storage and Testing]
A new nofuzz go build tag now disables gofuzz support. Release binaries enable this. (kubernetes/kubernetes#92491, @BenTheElder) [SIG API Machinery]
Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest (kubernetes/kubernetes#95741, @katiewasnothere) [SIG Node]
Add a serving and terminating condition to the EndpointSlice API.
serving tracks the readiness of endpoints regardless of their terminating state. This is distinct from ready since ready is only true when pods are not terminating.
terminating is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. (kubernetes/kubernetes#92968, @andrewsykim) [SIG Apps and Network]
Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API.
It changes the dual-stack API wrt Service from a single ipFamily field to 3
fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack),
ipFamilies (a list of families assigned), and clusterIPs (inclusive of
clusterIP). Most users do not need to set anything at all, defaulting will
handle it for them. Services are single-stack unless the user asks for
dual-stack. This is all gated by the “IPv6DualStack” feature gate. (kubernetes/kubernetes#91824, @khenidak) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
Add support for hugepages to downward API (kubernetes/kubernetes#86102, @derekwaynecarr) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
Adds kubelet alpha feature, GracefulNodeShutdown which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. (kubernetes/kubernetes#96129, @bobbypage) [SIG Node]
AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. (kubernetes/kubernetes#96327, @robscott) [SIG Apps and Network]
Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter
Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. (kubernetes/kubernetes#92744, @uablrek) [SIG Apps and Network]
Certain fields on Service objects will be automatically cleared when changing the service’s type to a mode that does not need those fields. For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. (kubernetes/kubernetes#95196, @thockin) [SIG API Machinery, Apps, Network and Testing]
Document that ServiceTopology feature is required to use service.spec.topologyKeys. (kubernetes/kubernetes#96528, @andrewsykim) [SIG Apps]
EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate.
- EndpointSlice topology field will be deprecated in an upcoming release.
- EndpointSlice “IP” address type is formally removed after being deprecated in Kubernetes 1.17.
- The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. (kubernetes/kubernetes#96440, @robscott) [SIG API Machinery, Apps and Network]
External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ (kubernetes/kubernetes#92632, @RenaudWasTaken) [SIG Node and Testing]
Fewer candidates are enumerated for preemption to improve performance in large clusters. (kubernetes/kubernetes#94814, @adtac)
Fix conversions for custom metrics. (kubernetes/kubernetes#94481, @wojtek-t) [SIG API Machinery and Instrumentation]
GPU metrics provided by kubelet are now disabled by default. (kubernetes/kubernetes#95184, @RenaudWasTaken)
If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric serviceaccount_stale_tokens_total to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting kube-apiserver with flag --service-account-extend-token-expiration=false (kubernetes/kubernetes#96273, @zshihang) [SIG API Machinery and Auth]
Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. (kubernetes/kubernetes#94196, @andrewsykim) [SIG Node and Release]
Introduces a metric source for HPAs which allows scaling based on container resource usage. (kubernetes/kubernetes#90691, @arjunrn) [SIG API Machinery, Apps, Autoscaling and CLI]
Kube-apiserver now deletes expired kube-apiserver Lease objects:
- The feature is under feature gate
APIServerIdentity.
- A flag is added to kube-apiserver:
identity-lease-garbage-collection-check-period-seconds (kubernetes/kubernetes#95895, @roycaihw) [SIG API Machinery, Apps, Auth and Testing]
Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting --volume-host-allow-local-loopback=false, or from contacting specific CIDR ranges by setting --volume-host-cidr-denylist (for example, --volume-host-cidr-denylist=127.0.0.1/28,feed::/16) (kubernetes/kubernetes#91785, @mattcary) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock (kubernetes/kubernetes#94603, @wojtek-t) [SIG API Machinery, Apps, Cloud Provider and Scheduling]
Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly (kubernetes/kubernetes#94182, @mattfenwick) [SIG API Machinery, Apps, Auth, Network and Node]
Move configurable fsgroup change policy for pods to beta (kubernetes/kubernetes#96376, @gnufied) [SIG Apps and Storage]
New flag is introduced, i.e. –topology-manager-scope=container|pod.
The default value is the “container” scope. (kubernetes/kubernetes#92967, @cezaryzukowski) [SIG Instrumentation, Node and Testing]
New parameter defaultingType for PodTopologySpread plugin allows to use k8s defined or user provided default constraints (kubernetes/kubernetes#95048, @alculquicondor) [SIG Scheduling]
NodeAffinity plugin can be configured with AddedAffinity. (kubernetes/kubernetes#96202, @alculquicondor) [SIG Node, Scheduling and Testing]
Promote RuntimeClass feature to GA.
Promote node.k8s.io API groups from v1beta1 to v1. (kubernetes/kubernetes#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]
Reminder: The labels “failure-domain.beta.kubernetes.io/zone” and “failure-domain.beta.kubernetes.io/region” are deprecated in favor of “topology.kubernetes.io/zone” and “topology.kubernetes.io/region” respectively. All users of the “failure-domain.beta…” labels should switch to the “topology…” equivalents. (kubernetes/kubernetes#96033, @thockin) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing]
Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. (kubernetes/kubernetes#93901, @jpbetz) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
Services will now have a clusterIPs field to go with clusterIP. clusterIPs[0] is a synonym for clusterIP and will be synchronized on create and update operations. (kubernetes/kubernetes#95894, @thockin) [SIG Network]
The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. (kubernetes/kubernetes#91921, @mtaufen) [SIG Auth]
The status of v1beta1 CRDs without “preserveUnknownFields:false” now shows a violation, “spec.preserveUnknownFields: Invalid value: true: must be false”. (kubernetes/kubernetes#93078, @vareti)
The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBService is enabled. The feature gate is disabled by default. The user has to enable it for the API Server. (kubernetes/kubernetes#94028, @janosi) [SIG API Machinery and Apps]
This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in CSIDriverSpec. (kubernetes/kubernetes#93130, @zshihang) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
Users can try the CronJob controller v2 using the feature gate. This will be the default controller in future releases. (kubernetes/kubernetes#93370, @alaypatel07) [SIG API Machinery, Apps, Auth and Testing]
VolumeSnapshotDataSource moves to GA in 1.20 release (kubernetes/kubernetes#95282, @xing-yang) [SIG Apps]
WinOverlay feature graduated to beta (kubernetes/kubernetes#94807, @ksubrmnn) [SIG Windows]
API priority and fairness graduated to beta
1.19 servers with APF turned on should not be run in a multi-server cluster with 1.20+ servers. (kubernetes/kubernetes#96527, @adtac) [SIG API Machinery and Testing]
Add LoadBalancerIPMode feature gate (kubernetes/kubernetes#92312, @Sh4d1) [SIG Apps, CLI, Cloud Provider and Network]
Add WindowsContainerResources and Annotations to CRI-API UpdateContainerResourcesRequest (kubernetes/kubernetes#95741, @katiewasnothere) [SIG Node]
Add a ‘serving’ and terminating condition to the EndpointSlice API.
serving tracks the readiness of endpoints regardless of their terminating state. This is distinct from ready since ready is only true when pods are not terminating.
terminating is true when an endpoint is terminating. For pods this is any endpoint with a deletion timestamp. (kubernetes/kubernetes#92968, @andrewsykim) [SIG Apps and Network]
Add support for hugepages to downward API (kubernetes/kubernetes#86102, @derekwaynecarr) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
Adds kubelet alpha feature, GracefulNodeShutdown which makes kubelet aware of node system shutdowns and result in graceful termination of pods during a system shutdown. (kubernetes/kubernetes#96129, @bobbypage) [SIG Node]
AppProtocol is now GA for Endpoints and Services. The ServiceAppProtocol feature gate will be deprecated in 1.21. (kubernetes/kubernetes#96327, @robscott) [SIG Apps and Network]
Automatic allocation of NodePorts for services with type LoadBalancer can now be disabled by setting the (new) parameter
Service.spec.allocateLoadBalancerNodePorts=false. The default is to allocate NodePorts for services with type LoadBalancer which is the existing behavior. (kubernetes/kubernetes#92744, @uablrek) [SIG Apps and Network]
Document that ServiceTopology feature is required to use service.spec.topologyKeys. (kubernetes/kubernetes#96528, @andrewsykim) [SIG Apps]
EndpointSlice has a new NodeName field guarded by the EndpointSliceNodeName feature gate.
- EndpointSlice topology field will be deprecated in an upcoming release.
- EndpointSlice “IP” address type is formally removed after being deprecated in Kubernetes 1.17.
- The discovery.k8s.io/v1alpha1 API is deprecated and will be removed in Kubernetes 1.21. (kubernetes/kubernetes#96440, @robscott) [SIG API Machinery, Apps and Network]
Fewer candidates are enumerated for preemption to improve performance in large clusters (kubernetes/kubernetes#94814, @adtac) [SIG Scheduling]
If BoundServiceAccountTokenVolume is enabled, cluster admins can use metric serviceaccount_stale_tokens_total to monitor workloads that are depending on the extended tokens. If there are no such workloads, turn off extended tokens by starting kube-apiserver with flag --service-account-extend-token-expiration=false (kubernetes/kubernetes#96273, @zshihang) [SIG API Machinery and Auth]
Introduce alpha support for exec-based container registry credential provider plugins in the kubelet. (kubernetes/kubernetes#94196, @andrewsykim) [SIG Node and Release]
Kube-apiserver now deletes expired kube-apiserver Lease objects:
- The feature is under feature gate
APIServerIdentity.
- A flag is added to kube-apiserver:
identity-lease-garbage-collection-check-period-seconds (kubernetes/kubernetes#95895, @roycaihw) [SIG API Machinery, Apps, Auth and Testing]
Move configurable fsgroup change policy for pods to beta (kubernetes/kubernetes#96376, @gnufied) [SIG Apps and Storage]
New flag is introduced, i.e. –topology-manager-scope=container|pod.
The default value is the “container” scope. (kubernetes/kubernetes#92967, @cezaryzukowski) [SIG Instrumentation, Node and Testing]
NodeAffinity plugin can be configured with AddedAffinity. (kubernetes/kubernetes#96202, @alculquicondor) [SIG Node, Scheduling and Testing]
Promote RuntimeClass feature to GA.
Promote node.k8s.io API groups from v1beta1 to v1. (kubernetes/kubernetes#95718, @SergeyKanzhelev) [SIG Apps, Auth, Node, Scheduling and Testing]
Reminder: The labels “failure-domain.beta.kubernetes.io/zone” and “failure-domain.beta.kubernetes.io/region” are deprecated in favor of “topology.kubernetes.io/zone” and “topology.kubernetes.io/region” respectively. All users of the “failure-domain.beta…” labels should switch to the “topology…” equivalents. (kubernetes/kubernetes#96033, @thockin) [SIG API Machinery, Apps, CLI, Cloud Provider, Network, Node, Scheduling, Storage and Testing]
The usage of mixed protocol values in the same LoadBalancer Service is possible if the new feature gate MixedProtocolLBSVC is enabled.
“action required”
The feature gate is disabled by default. The user has to enable it for the API Server. (kubernetes/kubernetes#94028, @janosi) [SIG API Machinery and Apps]
This PR will introduce a feature gate CSIServiceAccountToken with two additional fields in CSIDriverSpec. (kubernetes/kubernetes#93130, @zshihang) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
Users can try the CronJob controller v2 using the feature gate. This will be the default controller in future releases. (kubernetes/kubernetes#93370, @alaypatel07) [SIG API Machinery, Apps, Auth and Testing]
VolumeSnapshotDataSource moves to GA in 1.20 release (kubernetes/kubernetes#95282, @xing-yang) [SIG Apps]
-
Certain fields on Service objects will be automatically cleared when changing the service’s type to a mode that does not need those fields. For example, changing from type=LoadBalancer to type=ClusterIP will clear the NodePort assignments, rather than forcing the user to clear them. (kubernetes/kubernetes#95196, @thockin) [SIG API Machinery, Apps, Network and Testing]
Services will now have a clusterIPs field to go with clusterIP. clusterIPs[0] is a synonym for clusterIP and will be synchronized on create and update operations. (kubernetes/kubernetes#95894, @thockin) [SIG Network]
Add dual-stack Services (alpha). This is a BREAKING CHANGE to an alpha API.
It changes the dual-stack API wrt Service from a single ipFamily field to 3
fields: ipFamilyPolicy (SingleStack, PreferDualStack, RequireDualStack),
ipFamilies (a list of families assigned), and clusterIPs (inclusive of
clusterIP). Most users do not need to set anything at all, defaulting will
handle it for them. Services are single-stack unless the user asks for
dual-stack. This is all gated by the “IPv6DualStack” feature gate. (kubernetes/kubernetes#91824, @khenidak) [SIG API Machinery, Apps, CLI, Network, Node, Scheduling and Testing]
Introduces a metric source for HPAs which allows scaling based on container resource usage. (kubernetes/kubernetes#90691, @arjunrn) [SIG API Machinery, Apps, Autoscaling and CLI]
New parameter defaultingType for PodTopologySpread plugin allows to use k8s defined or user-provided default constraints (kubernetes/kubernetes#95048, @alculquicondor) [SIG Scheduling]
GPU metrics provided by kubelet are now disabled by default (kubernetes/kubernetes#95184, @RenaudWasTaken) [SIG Node]
New parameter defaultingType for PodTopologySpread plugin allows to use k8s defined or user provided default constraints (kubernetes/kubernetes#95048, @alculquicondor) [SIG Scheduling]
Server Side Apply now treats LabelSelector fields as atomic (meaning the entire selector is managed by a single writer and updated together), since they contain interrelated and inseparable fields that do not merge in intuitive ways. (kubernetes/kubernetes#93901, @jpbetz) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
Status of v1beta1 CRDs without “preserveUnknownFields:false” will show violation “spec.preserveUnknownFields: Invalid value: true: must be false” (kubernetes/kubernetes#93078, @vareti) [SIG API Machinery]
A new nofuzz go build tag now disables gofuzz support. Release binaries enable this. (kubernetes/kubernetes#92491, @BenTheElder) [SIG API Machinery]
A new alpha-level field, SupportsFsGroup, has been introduced for CSIDrivers to allow them to specify whether they support volume ownership and permission modifications. The CSIVolumeSupportFSGroup feature gate must be enabled to allow this field to be used. (kubernetes/kubernetes#92001, @huffmanca) [SIG API Machinery, CLI and Storage]
Added pod version skew strategy for seccomp profile to synchronize the deprecated annotations with the new API Server fields. Please see the corresponding section in the KEP for more detailed explanations. (kubernetes/kubernetes#91408, @saschagrunert) [SIG Apps, Auth, CLI and Node]
Adds the ability to disable Accelerator/GPU metrics collected by Kubelet (kubernetes/kubernetes#91930, @RenaudWasTaken) [SIG Node]
Custom Endpoints are now mirrored to EndpointSlices by a new EndpointSliceMirroring controller. (kubernetes/kubernetes#91637, @robscott) [SIG API Machinery, Apps, Auth, Cloud Provider, Instrumentation, Network and Testing]
External facing API podresources is now available under k8s.io/kubelet/pkg/apis/ (kubernetes/kubernetes#92632, @RenaudWasTaken) [SIG Node and Testing]
Fix conversions for custom metrics. (kubernetes/kubernetes#94481, @wojtek-t) [SIG API Machinery and Instrumentation]
Generic ephemeral volumes, a new alpha feature under the GenericEphemeralVolume feature gate, provide a more flexible alternative to EmptyDir volumes: as with EmptyDir, volumes are created and deleted for each pod automatically by Kubernetes. But because the normal provisioning process is used (PersistentVolumeClaim), storage can be provided by third-party storage vendors and all of the usual volume features work. Volumes don’t need to be empty; for example, restoring from snapshot is supported. (kubernetes/kubernetes#92784, @pohly) [SIG API Machinery, Apps, Auth, CLI, Instrumentation, Node, Scheduling, Storage and Testing]
Kube-controller-manager: volume plugins can be restricted from contacting local and loopback addresses by setting --volume-host-allow-local-loopback=false, or from contacting specific CIDR ranges by setting --volume-host-cidr-denylist (for example, --volume-host-cidr-denylist=127.0.0.1/28,feed::/16) (kubernetes/kubernetes#91785, @mattcary) [SIG API Machinery, Apps, Auth, CLI, Network, Node, Storage and Testing]
Kubernetes is now built with golang 1.15.0-rc.1.
- The deprecated, legacy behavior of treating the CommonName field on X.509 serving certificates as a host name when no Subject Alternative Names are present is now disabled by default. It can be temporarily re-enabled by adding the value x509ignoreCN=0 to the GODEBUG environment variable. (kubernetes/kubernetes#93264, @justaugustus) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scalability, Storage and Testing]
Migrate scheduler, controller-manager and cloud-controller-manager to use LeaseLock (kubernetes/kubernetes#94603, @wojtek-t) [SIG API Machinery, Apps, Cloud Provider and Scheduling]
Modify DNS-1123 error messages to indicate that RFC 1123 is not followed exactly (kubernetes/kubernetes#94182, @mattfenwick) [SIG API Machinery, Apps, Auth, Network and Node]
The ServiceAccountIssuerDiscovery feature gate is now Beta and enabled by default. (kubernetes/kubernetes#91921, @mtaufen) [SIG Auth]
The kube-controller-manager managed signers can now have distinct signing certificates and keys. See the help about --cluster-signing-[signer-name]-{cert,key}-file. --cluster-signing-{cert,key}-file is still the default. (kubernetes/kubernetes#90822, @deads2k) [SIG API Machinery, Apps and Auth]
When creating a networking.k8s.io/v1 Ingress API object, spec.tls[*].secretName values are required to pass validation rules for Secret API object names. (kubernetes/kubernetes#93929, @liggitt) [SIG Network]
WinOverlay feature graduated to beta (kubernetes/kubernetes#94807, @ksubrmnn) [SIG Windows]
OpenAPI Generator has been updated to v4.3.0 from v3.3.4. Following are links to Python client related changes throughout the OpenAPI releases above v3.3.4 to v4.3.0:
Skipped because of a failed initial release.
