long running LT against a single pod on eks cluster #659
-
|
I never used Fortio before. Thanks, Vitaly. |
Beta Was this translation helpful? Give feedback.
Replies: 19 comments 28 replies
-
|
Are you using istio or something else on your k8s cluster? Accessing fortio's web UI is similar to accessing any other service or app. For instance using istio ingress: apiVersion: v1
kind: Namespace
metadata:
name: fortio
labels:
istio-injection: enabled
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: fortio
namespace: fortio
spec:
selector:
matchLabels:
app: fortio
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 0%
type: RollingUpdate
template:
metadata:
labels:
app: fortio
spec:
containers:
- name: fortio
image: fortio/fortio:latest_release
args:
- server
ports:
- containerPort: 8080
resources:
requests:
cpu: 2000m
---
apiVersion: v1
kind: Service
metadata:
name: fortio
namespace: fortio
labels:
app: fortio
service: fortio
spec:
ports:
- name: http2
port: 8080
selector:
app: fortio
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: fortio-gateway
namespace: fortio
spec:
selector:
istio: $INGRESS
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "fortio-$CLUSTER_ID.$REGION.$DOMAIN"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: echo
namespace: echo
spec:
hosts:
- "fortio-$CLUSTER_ID.$REGION.$DOMAIN"
gateways:
- fortio-gateway
http:
- match:
- headers:
host:
exact: "echo-$CLUSTER_ID.$REGION.$DOMAIN"
route:
- destination:
host: fortio.echo.svc.cluster.local
port:
number: 8080(in the above the namespace is fortio but can obviously be changed, not sure I'd put it in kube-system though) For request/limit likewise it depends what you're going to be asking that pod to do - fortio does scale very well/near linearly with more cores and consumes very little memory So if you wanted to generate >500k requests per second for instance (unlikely you need that many but...), you'd likely need at least 32-128 cores Once deployed you can use the web UI or the new rest endpoints to generate load (they take same url params like url= qps= t= (time) etc... which mostly match the flag names you see running fortio help) I haven't had to use QoS personally but it's an interesting thought |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your reply.
We are not using istio on this k8s non prod cluster.
Vitaly.
…On Wed, Nov 16, 2022 at 2:28 PM Laurent Demailly ***@***.***> wrote:
Are you using istio or something else on your k8s cluster?
Accessing fortio's web UI is similar to accessing any other service or
app. For instance using istio ingress:
apiVersion: v1kind: Namespacemetadata:
name: fortio
labels:
istio-injection: enabled
---apiVersion: apps/v1kind: Deploymentmetadata:
name: fortio
namespace: fortiospec:
selector:
matchLabels:
app: fortio
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 0%
type: RollingUpdate
template:
metadata:
labels:
app: fortio
spec:
containers:
- name: fortio
image: fortio/fortio:latest_release
args:
- server
- -ui-path
- ""
ports:
- containerPort: 8080
resources:
requests:
cpu: 2000m
---apiVersion: v1kind: Servicemetadata:
name: fortio
namespace: fortio
labels:
app: fortio
service: fortiospec:
ports:
- name: http2
port: 8080
selector:
app: fortio
---apiVersion: networking.istio.io/v1alpha3kind: Gatewaymetadata:
name: fortio-gateway
namespace: fortiospec:
selector:
istio: $INGRESS
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "fortio-$CLUSTER_ID.$REGION.$DOMAIN"
---apiVersion: networking.istio.io/v1alpha3kind: VirtualServicemetadata:
name: echo
namespace: echospec:
hosts:
- "fortio-$CLUSTER_ID.$REGION.$DOMAIN"
gateways:
- fortio-gateway
http:
- match:
- headers:
host:
exact: "echo-$CLUSTER_ID.$REGION.$DOMAIN"
route:
- destination:
host: fortio.echo.svc.cluster.local
port:
number: 8080
(in the above the namespace is fortio but can obviously be changed, not
sure I'd put it in kube-system though)
For request/limit likewise it depends what you're going to be asking that
pod to do - fortio does scale very well/near linearly with more cores and
consumes very little memory
So if you wanted to generate >500k requests per second for instance
(unlikely you need that many but...), you'd likely need at least 32-128
cores
Once deployed you can use the web UI or the new rest endpoints to generate
load (they take same url params like url= qps= t= (time) etc... which
mostly match the flag names you see running fortio help)
I haven't had to use QoS personally but it's an interesting thought
—
Reply to this email directly, view it on GitHub
<#659 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFJNLXQETTZSTZLUY6JS7R3WIVNZBANCNFSM6AAAAAASCJTUYM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Then use whichever way you would expose any other web service (either NLB or other)? |
Beta Was this translation helpful? Give feedback.
-
|
We got fortio v.1.38.4 control UI.
1. After we started LT we observed that the fortio pod/container were
receiving some traffic, but the endpoint
did not, based on the info from kubectl logs (pod) -n (namespace).
2. When the Duration was set to 30s, no graphical output was provided.
We need your help.
[image: image.png]
…On Thu, Nov 17, 2022 at 7:27 PM Laurent Demailly ***@***.***> wrote:
Then use whichever way you would expose any other web service (either NLB
or other)?
—
Reply to this email directly, view it on GitHub
<#659 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFJNLXRPIOEBCDT5WELN7KTWI3ZSTANCNFSM6AAAAAASCJTUYM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Screenshot is missing Also send the logs (kubectl logs …fortio’s pod…) |
Beta Was this translation helpful? Give feedback.
-
|
18:17:45 I httprunner.go:102> Starting http test for http://localhost:8080/echo?delay=250us:30,5ms:5&status=503:0.5,429:1.5 with 30 threads at 3000.0 qps and parallel warmup |
Beta Was this translation helpful? Give feedback.
-
|
However, there was no calls to endpoints |
Beta Was this translation helpful? Give feedback.
-
|
Please let me know what else you need. |
Beta Was this translation helpful? Give feedback.
-
|
You left the demo url to exercise unchanged (localhost 8080) so fortio just called itself. If you want it to call something else, change the url |
Beta Was this translation helpful? Give feedback.
-
|
Thanks,
I change URL for the endpoins service.
However. the logs for the endpoints does not show any coming requests,
…On Sun, Nov 20, 2022 at 3:18 AM Laurent Demailly ***@***.***> wrote:
You left the demo url to exercise unchanged (localhost 8080) so fortio
just called itself. If you want it to call something else, change the url
—
Reply to this email directly, view it on GitHub
<#659 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFJNLXX4YI6227Y42QHWTZ3WJICKFANCNFSM6AAAAAASCJTUYM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Send the log or output from fortio - are there errors? |
Beta Was this translation helpful? Give feedback.
-
|
20:41:13 UI: GET /fortio/ HTTP/1.1 100.118.238.105:54728 (https) 63.15.0.104 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chro me/107.0.0.0 Safari/537.36" |
Beta Was this translation helpful? Give feedback.
-
|
This is the log from the endpoint. |
Beta Was this translation helpful? Give feedback.
-
|
100% errors |
Beta Was this translation helpful? Give feedback.
-
|
you left the fortio demo query args. make sure the url you're testing works, first. you can try any url as single query first in the section below the load test one. or use kubectl exec fortio curl |
Beta Was this translation helpful? Give feedback.
-
|
and the pod logs would have details about what the error was |
Beta Was this translation helpful? Give feedback.
-
|
Starting at 10 qps with 8 thread(s) [gomax 32] for 3s : 3 calls each (total 24) range, mid point, percentile, count
target 50% 0.0016target 75% 0.002125target 90% 0.00265target 99% 0.00302913target 99.9% 0.00303741Error cases : count 24 avg 0.0017011282 +/- 0.0006459 min 0.000830986 max 0.003038335 sum 0.040827077 range, mid point, percentile, count
target 50% 0.0016target 75% 0.002125target 90% 0.00265target 99% 0.00302913target 99.9% 0.00303741Socket and IP used for each connection:[0] 4 socket used, resolved to [172.20.30.108:8080] connection timing : count 4 avg 0.00070026275 +/- 0.0002159 min 0.000329702 max 0.000855934 sum 0.002801051 range, mid point, percentile, count
target 50% 0.000575472target 75% 0.00072923target 90% 0.000821486target 99% 0.000876839target 99.9% 0.000882374Sockets used: 32 (for perfect keepalive, would be 8) |
Beta Was this translation helpful? Give feedback.
-
|
Φορτίο (fortio) v1.38.4 Fortio http://172.20.65.95:8080/ev6v/nssit4/browsing/actuator/info edit: use 3 ` to avoid markdown formatting of > etc |
Beta Was this translation helpful? Give feedback.
-
|
logs from one pod: |
Beta Was this translation helpful? Give feedback.
-
|
I meant the fortio pod but it seems your application/target also is complaining/having errors again start by having 1 single successful curl before trying to do tens of thousands of the same |
Beta Was this translation helpful? Give feedback.
-
|
anything wrong? |
Beta Was this translation helpful? Give feedback.
-
|
yes when it says WARNING 100.00% of sleep were falling behind it means your qps can't be achieved. see the faq I linked below/in previous answer |
Beta Was this translation helpful? Give feedback.
-
|
This ^, again (and my other replies too) |
Beta Was this translation helpful? Give feedback.
-
|
I doubled the number of qps and no results coming. |
Beta Was this translation helpful? Give feedback.
-
|
it's rare that if you are having errors at one qps level, it gets better by doubling it |
Beta Was this translation helpful? Give feedback.
-
|
please also read https://github.com/fortio/fortio/wiki/FAQ#i-want-to-get-the-best-results-what-flags-should-i-pass specifically you need to have enough connections to reach a qps given the latency / rountrip to the service you are testing |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
went back to 10 connections. almost no errors. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
What to do to increase the pod cpu% over 90% to get HPA to allocate more pods |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Much better results with 8 connections.
|
Beta Was this translation helpful? Give feedback.
-
|
Sockets used: 8 (for perfect keepalive, would be 8) |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
what 130 calls mean. where those calls were going. based on 30 seconds interval it arrives at qps = 4.1602. I specified 4500 qps over 30 seconds. Please explain. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
CPU % for the application stays the same. Does not depend on qps. What to do? |
Beta Was this translation helpful? Give feedback.
-
|
no increase in real qps.
|
Beta Was this translation helpful? Give feedback.
-
|
the last test tonight.
|
Beta Was this translation helpful? Give feedback.
-
|
Memory usage is high. But there is no memory threshold for HPA to schedule more pods. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
It means what it says, let me decode:
Means it only did 131 calls for the duration requested, because the average response time was 1.9 seconds, so it reached only 4.1 requests per second despite whatever you asked. (this is covered in the faq on how latency and qps and number of connections relate) |
Beta Was this translation helpful? Give feedback.
-
|
Φορτίο (fortio) v1.38.4 Fortio 15:00 pm; 9000 qps over 90 s; uniform; 8 connections http://172.20.65.95:8080/ev6v/nssit4/browsing/actuator/info Running load test... Results pending... Interrupt |
Beta Was this translation helpful? Give feedback.
-
|
Results pending for how long? |
Beta Was this translation helpful? Give feedback.
-
|
Not sure how many times I have to ask you to read the faq or repeat the same thing about not increasing qps before you get the smaller numbers working and understood. You can't do 9000 qps with 8 threads when your service takes 2s to reply to each request (in case it's not obvious you can only do 0.5 request/sec per thread/connection with that latency... you would need 18,000 connections which wouldn't work) Again: https://github.com/fortio/fortio/wiki/FAQ#i-want-to-get-the-best-results-what-flags-should-i-pass I'm going to close this thread as I exhausted my quota for repetition. Happy thanksgiving and best of luck |
Beta Was this translation helpful? Give feedback.
Are you using istio or something else on your k8s cluster?
Accessing fortio's web UI is similar to accessing any other service or app. For instance using istio ingress: