Load testing microservice - bad results
I am struggling with the results of a load test of my api service.
The service is deployed on GCP. There are 2 instances running behind a nginx.
I am using spring boot 2.0 with default tomcat configuration.
The service is running inside a docker container with -Xmx768M
A request to the tested endpoint fetches a single row from a database and returns it.
My results:
ab -c 1 -n 1 /testapi
Requests per second: 15
Time per request: 63ms
ab -c 10 -n 100 /testapi
Requests per second: 102
Time per request: 97ms
ab -c 50 -n 100 /testapi
Requests per second: 95
Time per request: 522ms
ab -c 100 -n 100 /testapi
Requests per second: 93
Time per request: 1065ms
ab -c 200 -n 200 /testapi
Timeout
Worrying aspects:
Heap usage is rising after each test
At 99.99% heap usage I am getting slow responses until the container crashes with java.lang.OutOfMemoryError: Java heap space
I am able to ddos 2 instances of an API from a single laptop.
What am I doing wrong here? Does the tomcat configuration needs further tweaking? Are the JVM parameters wrong?
Thanks
EDIT tested code:
@ApiOperation(value="Get Device", response = Device.class, authorizations = {@Authorization(value = "Authorization")})
@RequestMapping(value="/device/commnr/{commnr}", method= RequestMethod.GET)
public DeferredResult<ResponseEntity> getDevice(@PathVariable String commnr) {
final DeferredResult<ResponseEntity> deferred = new DeferredResult<>();
HttpServletRequest request = getCurrentHttpRequest();
timer.schedule(new TimerTask() {
@Override
public void run() {
if(deferred.isSetOrExpired()) {
throw new RuntimeException();
} else {
User user = authenticationService.loginUser(request);
Device device = deviceService.findByCommNr(commnr);
if(!AuthTools.isUserDevice(device, user)) {
deferred.setResult(new ResponseEntity<>("Requested Device does not belong to user.", HttpStatus.FORBIDDEN));
}
if(device == null) {
deferred.setResult(new ResponseEntity<>(HttpStatus.NOT_FOUND));
}
deferred.setResult(new ResponseEntity<>(device, HttpStatus.OK));
}
}
}, 10);
return deferred;
}
performance spring-boot tomcat google-cloud-platform load-testing
asked Nov 20 '18 at 17:06
Alex TbkAlex Tbk
517522
add a comment |
I am struggling with the results of a load test of my api service.
The service is deployed on GCP. There are 2 instances running behind a nginx.
I am using spring boot 2.0 with default tomcat configuration.
The service is running inside a docker container with -Xmx768M
A request to the tested endpoint fetches a single row from a database and returns it.
My results:
ab -c 1 -n 1 /testapi
Requests per second: 15
Time per request: 63ms
ab -c 10 -n 100 /testapi
Requests per second: 102
Time per request: 97ms
ab -c 50 -n 100 /testapi
Requests per second: 95
Time per request: 522ms
ab -c 100 -n 100 /testapi
Requests per second: 93
Time per request: 1065ms
ab -c 200 -n 200 /testapi
Timeout
Worrying aspects:
Heap usage is rising after each test
At 99.99% heap usage I am getting slow responses until the container crashes with java.lang.OutOfMemoryError: Java heap space
I am able to ddos 2 instances of an API from a single laptop.
What am I doing wrong here? Does the tomcat configuration needs further tweaking? Are the JVM parameters wrong?
Thanks
EDIT tested code:
@ApiOperation(value="Get Device", response = Device.class, authorizations = {@Authorization(value = "Authorization")})
@RequestMapping(value="/device/commnr/{commnr}", method= RequestMethod.GET)
public DeferredResult<ResponseEntity> getDevice(@PathVariable String commnr) {
final DeferredResult<ResponseEntity> deferred = new DeferredResult<>();
HttpServletRequest request = getCurrentHttpRequest();
timer.schedule(new TimerTask() {
@Override
public void run() {
if(deferred.isSetOrExpired()) {
throw new RuntimeException();
} else {
User user = authenticationService.loginUser(request);
Device device = deviceService.findByCommNr(commnr);
if(!AuthTools.isUserDevice(device, user)) {
deferred.setResult(new ResponseEntity<>("Requested Device does not belong to user.", HttpStatus.FORBIDDEN));
}
if(device == null) {
deferred.setResult(new ResponseEntity<>(HttpStatus.NOT_FOUND));
}
deferred.setResult(new ResponseEntity<>(device, HttpStatus.OK));
}
}
}, 10);
return deferred;
}
performance spring-boot tomcat google-cloud-platform load-testing
asked Nov 20 '18 at 17:06
Alex TbkAlex Tbk
517522
Are you using Spring WebFlux?
– Boris
Nov 20 '18 at 17:08
I added the code from the tested method.
– Alex Tbk
Nov 20 '18 at 17:12
You should take and analyze a memory snapshot using tools like Memory Analyzer (MAT) or VisualVM. It's kind of hard to guess which class instances are leaking.
– Selaron
Nov 20 '18 at 19:04
add a comment |
I am struggling with the results of a load test of my api service.
The service is deployed on GCP. There are 2 instances running behind a nginx.
I am using spring boot 2.0 with default tomcat configuration.
The service is running inside a docker container with -Xmx768M
A request to the tested endpoint fetches a single row from a database and returns it.
My results:
ab -c 1 -n 1 /testapi
Requests per second: 15
Time per request: 63ms
ab -c 10 -n 100 /testapi
Requests per second: 102
Time per request: 97ms
ab -c 50 -n 100 /testapi
Requests per second: 95
Time per request: 522ms
ab -c 100 -n 100 /testapi
Requests per second: 93
Time per request: 1065ms
ab -c 200 -n 200 /testapi
Timeout
Worrying aspects:
Heap usage is rising after each test
At 99.99% heap usage I am getting slow responses until the container crashes with java.lang.OutOfMemoryError: Java heap space
I am able to ddos 2 instances of an API from a single laptop.
What am I doing wrong here? Does the tomcat configuration needs further tweaking? Are the JVM parameters wrong?
Thanks
EDIT tested code:
@ApiOperation(value="Get Device", response = Device.class, authorizations = {@Authorization(value = "Authorization")})
@RequestMapping(value="/device/commnr/{commnr}", method= RequestMethod.GET)
public DeferredResult<ResponseEntity> getDevice(@PathVariable String commnr) {
final DeferredResult<ResponseEntity> deferred = new DeferredResult<>();
HttpServletRequest request = getCurrentHttpRequest();
timer.schedule(new TimerTask() {
@Override
public void run() {
if(deferred.isSetOrExpired()) {
throw new RuntimeException();
} else {
User user = authenticationService.loginUser(request);
Device device = deviceService.findByCommNr(commnr);
if(!AuthTools.isUserDevice(device, user)) {
deferred.setResult(new ResponseEntity<>("Requested Device does not belong to user.", HttpStatus.FORBIDDEN));
}
if(device == null) {
deferred.setResult(new ResponseEntity<>(HttpStatus.NOT_FOUND));
}
deferred.setResult(new ResponseEntity<>(device, HttpStatus.OK));
}
}
}, 10);
return deferred;
}
performance spring-boot tomcat google-cloud-platform load-testing
asked Nov 20 '18 at 17:06
Alex TbkAlex Tbk
517522
I am struggling with the results of a load test of my api service.
The service is deployed on GCP. There are 2 instances running behind a nginx.
I am using spring boot 2.0 with default tomcat configuration.
The service is running inside a docker container with -Xmx768M
A request to the tested endpoint fetches a single row from a database and returns it.
My results:
ab -c 1 -n 1 /testapi
Requests per second: 15
Time per request: 63ms
ab -c 10 -n 100 /testapi
Requests per second: 102
Time per request: 97ms
ab -c 50 -n 100 /testapi
Requests per second: 95
Time per request: 522ms
ab -c 100 -n 100 /testapi
Requests per second: 93
Time per request: 1065ms
ab -c 200 -n 200 /testapi
Timeout
Worrying aspects:
Heap usage is rising after each test
At 99.99% heap usage I am getting slow responses until the container crashes with java.lang.OutOfMemoryError: Java heap space
I am able to ddos 2 instances of an API from a single laptop.
What am I doing wrong here? Does the tomcat configuration needs further tweaking? Are the JVM parameters wrong?
Thanks
EDIT tested code:
@ApiOperation(value="Get Device", response = Device.class, authorizations = {@Authorization(value = "Authorization")})
@RequestMapping(value="/device/commnr/{commnr}", method= RequestMethod.GET)
public DeferredResult<ResponseEntity> getDevice(@PathVariable String commnr) {
final DeferredResult<ResponseEntity> deferred = new DeferredResult<>();
HttpServletRequest request = getCurrentHttpRequest();
timer.schedule(new TimerTask() {
@Override
public void run() {
if(deferred.isSetOrExpired()) {
throw new RuntimeException();
} else {
User user = authenticationService.loginUser(request);
Device device = deviceService.findByCommNr(commnr);
if(!AuthTools.isUserDevice(device, user)) {
deferred.setResult(new ResponseEntity<>("Requested Device does not belong to user.", HttpStatus.FORBIDDEN));
}
if(device == null) {
deferred.setResult(new ResponseEntity<>(HttpStatus.NOT_FOUND));
}
deferred.setResult(new ResponseEntity<>(device, HttpStatus.OK));
}
}
}, 10);
return deferred;
}
performance spring-boot tomcat google-cloud-platform load-testing
performance spring-boot tomcat google-cloud-platform load-testing
asked Nov 20 '18 at 17:06
Alex TbkAlex Tbk
517522
asked Nov 20 '18 at 17:06
Alex TbkAlex Tbk
517522
edited Nov 20 '18 at 17:11
Alex Tbk
asked Nov 20 '18 at 17:06
Alex TbkAlex Tbk
517522
asked Nov 20 '18 at 17:06
Alex TbkAlex Tbk
517522
asked Nov 20 '18 at 17:06
Alex TbkAlex Tbk
517522
517522
Are you using Spring WebFlux?
– Boris
Nov 20 '18 at 17:08
I added the code from the tested method.
– Alex Tbk
Nov 20 '18 at 17:12
You should take and analyze a memory snapshot using tools like Memory Analyzer (MAT) or VisualVM. It's kind of hard to guess which class instances are leaking.
– Selaron
Nov 20 '18 at 19:04
add a comment |
Are you using Spring WebFlux?
– Boris
Nov 20 '18 at 17:08
I added the code from the tested method.
– Alex Tbk
Nov 20 '18 at 17:12
You should take and analyze a memory snapshot using tools like Memory Analyzer (MAT) or VisualVM. It's kind of hard to guess which class instances are leaking.
– Selaron
Nov 20 '18 at 19:04
Are you using Spring WebFlux?
– Boris
Nov 20 '18 at 17:08
Are you using Spring WebFlux?
– Boris
Nov 20 '18 at 17:08
I added the code from the tested method.
– Alex Tbk
Nov 20 '18 at 17:12
I added the code from the tested method.
– Alex Tbk
Nov 20 '18 at 17:12
You should take and analyze a memory snapshot using tools like Memory Analyzer (MAT) or VisualVM. It's kind of hard to guess which class instances are leaking.
– Selaron
Nov 20 '18 at 19:04
You should take and analyze a memory snapshot using tools like Memory Analyzer (MAT) or VisualVM. It's kind of hard to guess which class instances are leaking.
– Selaron
Nov 20 '18 at 19:04
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398036%2fload-testing-microservice-bad-results%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398036%2fload-testing-microservice-bad-results%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Are you using Spring WebFlux?
– Boris
Nov 20 '18 at 17:08
I added the code from the tested method.
– Alex Tbk
Nov 20 '18 at 17:12
You should take and analyze a memory snapshot using tools like Memory Analyzer (MAT) or VisualVM. It's kind of hard to guess which class instances are leaking.
– Selaron
Nov 20 '18 at 19:04