Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write an in-memory apiserver #1108

Open
clux opened this issue Dec 21, 2022 · 4 comments
Open

Write an in-memory apiserver #1108

clux opened this issue Dec 21, 2022 · 4 comments
Labels
automation ci and testing related help wanted Not immediately prioritised, please help! question Direction unclear; possibly a bug, possibly could be improved.

Comments

@clux
Copy link
Member

clux commented Dec 21, 2022 •
edited

What problem are you trying to solve?

A thing that continually comes up in the concept of controller testing is being able to run the reconciler and verify that it does the right thing.

In complex scenarios this is difficult for users to do right now without a semi-functioning apiserver.

We currently recommend using a mock client:

let (mock_service, handle) = tower_test::mock::pair::<Request<Body>, Response<Body>>();
let mock_client = Client::new(mock_service, "default");

and pass that into the reconciler's context where we intercept the api calls and return some reasonable information. See controller-rs's fixtures.rs and controller.rs for test invocations.

It is perfectly possible to do that in tests (and we do) like this particular wrapper around
tower_test::mock::Handle<Request<Body>, Response<Body>> responding to an Event
being POSTed, while also checking some properties of that data:

async fn handle_event_create(mut self, reason: String) -> Result<Self> {
    let (request, send) = self.0.next_request().await.expect("service not called");
    assert_eq!(request.method(), http::Method::POST);
    assert_eq!(
        request.uri().to_string(),
        format!("/apis/events.k8s.io/v1/namespaces/testns/events?")
    );
    // verify the event reason matches the expected
    let req_body = to_bytes(request.into_body()).await.unwrap();
    let postdata: serde_json::Value =
        serde_json::from_slice(&req_body).expect("valid event from runtime");
    dbg!("postdata for event: {}", postdata.clone());
    assert_eq!(
        postdata.get("reason").unwrap().as_str().map(String::from),
        Some(reason)
    );
    // then pass through the body
    send.send_response(Response::builder().body(Body::from(req_body)).unwrap());
    Ok(self)
}

The problem with this approach is that:

  • it is verbose (lots of Request / Body / Response / serde_json::Value fiddling)
  • it requires user tests implementing apiserver expected behavior
  • it mixes apiserver imitation behavior with test assertion logic

Describe the solution you'd like

Create a dumb, in-memory apiserver that does the bare minimum of what the apiserver does,
and presents a queryable interface that can give us what is in "its database" through some type downcasting.

This server could treat every object as a DynamicObject storing what it sees in a HashMap<ObjectRef, DynamicObject> as an initial memory backing.

If this was made pluggable into tower_test::mock, users can hook it into tests around a reconciler without the tests failing due to bad apiserver responses, without having to have users know all the ins and outs of apiserver mechanics (and crucially without giving them the opportunity to get this wrong).

Implementation

We would need at the very least implement basic functionality around metadata on objects:

  • POSTs need to fill in plausible creationTimestamp, uid, resourceVersion, generation, populate name from generateName
  • and prevent clients from overriding read-only values like creationTimestamp / uid / resource_version / generation
  • Respond to queries after storing the query result in the HashMap
  • DELETEs need to traverse ownerReferences

Implementing create/replace/delete/get calls on resources plus most calls on subresources "should not be too difficult" to do in this context and will benefit everyone.

The real problem here would be implementing patch in a sensible way:

  • json patches need to actually act on the dynamic object
  • apply patches need to follow kubernetes merge rules and actually do what the apiserver does
  • merge patches, strategic merges with patch strategies need to be followed

Some of this sounds very hard, but it's possible some of it can be cobbled together using existing ecosystem pieces like:

Anyway, just thought I would write down my thoughts on this. It feels possible, but certainly a bit of a spare time project. If anyone wants this, or would like to tackle this, let us know. I have too much on my plate for something like this right now, but personally I would love to have something like this if it can be done in a sane way.

Documentation, Adoption, Migration Strategy

can be integrated into controller-rs plus the controller guide on kube.rs as a start.
@clux clux added question Direction unclear; possibly a bug, possibly could be improved. help wanted Not immediately prioritised, please help! labels Dec 21, 2022
@clux
Copy link
Member Author

clux commented Dec 21, 2022

NB: Prior discussion around envtest #382 went down a separate route of relying on cluster provisioning to do more integration style tests. This is a heavy weight solution that either re-uses the test environment or sets up one cluster per test, so we currently have left this style of test automation to CI actions provisioning a test cluster.

It is currently pretty easy to do this for a small amount of integration tests. We use this approach, and it does not require mock clients, but it does force us to consider how some tests might interact with others (limiting how deeply we can use this approach). This issue is instead trying to turn more complicated integration tests into true unit tests for users. Both style of tests have a place.

@ianstanton
Copy link
Contributor

cc @ChuckHend @sjmiller609

@clux clux mentioned this issue Jan 4, 2023
Closed
@clux clux mentioned this issue Mar 5, 2023
Closed
@clux
Copy link
Member Author

clux commented Mar 5, 2023 •
edited

A perhaps more promising, and less work-involving way forward here for integration tests is to lean on; https://github.com/kubernetes-sigs/kwok/

kwok still creates some kind of cluster, but it seems to mock out a lot of the more node/pod related behaviour, and as such it could become a slightly more reliable integration method / serve as a de-facto mock server (that's kind of a real apiserver).

We have not done any particular testing of this yet (and it's currently in a super early release), but noting this down.
If you are doing any experiments with kwok for kube please share!

Have closed our older kube-test / envtest proposals in the past.

@clux clux added the automation ci and testing related label Mar 5, 2023
@Bakrog
Copy link

Bakrog commented Apr 26, 2024

Thanks for the suggestion on using KWOK @clux. It's working really well on my tests, and I spin up multiple clusters of it using testcontainers. If you would like I could create a PR adding documentation about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
automation ci and testing related help wanted Not immediately prioritised, please help! question Direction unclear; possibly a bug, possibly could be improved.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@clux @Bakrog @ianstanton