Eliminate the global state RayJob reconciler

[mimowo]

What would you like to be cleaned:

The current WIP PR #8341 is introducing the global state, following the pattern for TrainJob, but we should fix it going forward also for TrainJob: #9021

Why is this needed:

To avoid non-obvious failures in MultiKueue integration tests, motivation as for #9021

[mimowo]

@hiboyang would you maybe find cycles to take the issue as you have already all the context? At the technical level I'm considering two options:

1. extend the PodSets interface
2. store the client inside the reconciler instance and allow to retrieve it via JobReconcilerInterface.

I think it makes sense to go for (2.) because it already has SetupWithManager and the lifetime of "client" is the same as "manager". Also, we will then have the client available in all functions, not just "PodSets" for free, wdyt?

[hiboyang]

Sounds good, I will work on this, will take a look at JobReconcilerInterface. Thanks for the pointer!

[hiboyang]

@mimowo I am looking at JobReconcilerInterface, it is possible to pass a "client" to the reconciler by extending SetupWithManager method, but we still need some way to pass the "client" to GenericJob.PodSets(ctx context.Context), which does not have reference to the reconciler.

How do you like the idea to create another interface like GenericJobV2 (could be other name), GenericJobV2 extends GenericJob with an extra method FetchPodSets(ctx context.Context, c client.Client). RayJob will implement GenericJobV2. This will not break backward compatibility.

Or add a new interface (similar like other single method interface like JobWithPodLabelSelector):

type JobWithFetchPodSets interface {
	FetchPodSets(ctx context.Context, c client.Client) ([]kueue.PodSet, error)
}

[mimowo]

I think JobWithFetchPodSets is an unnecessary complication.

I'm ok with extending the PodSets interface for 0.17, unless maybe @tenzen-y has a better idea. Or maybe @kaisoz who work on TrainJob which also have the global state issue.

[tenzen-y]

Sorry I have no context for this issue. Could you point the root cause the reason why eliminating RayJob reconciller? Does that mean that you want to follow the similar pattern with LWS / StatefulSet?

[mimowo]

Not the reconciler as such, just the global state: https://github.com/kubernetes-sigs/kueue/blob/main/pkg/controller/jobs/trainjob/trainjob_controller.go#L87