Slow I/O on VM hosts
I am currently debugging guests of a host of mine that is suffering from high latency but acceptable read/write speeds on CentOS 6. This VM is connected to a VM host which has its storage on a SAN that I am reading/writing to. 7K/10K RPM HDDs.
Here is a test output (CentOS - LSI Logic SAS):
1073741824 bytes (1.1 GB) copied, 10.4863 s, 102 MB/s
Here is another test output (Windows - same SAN, different volume, VMWare Paravirtual, updated tools):
Read IO 9MiB/s, 1152 I/O per s, 582ms latency on last I/O request
Here is yet another test ouput from a machine with direct access to SAN:
Read IO 90MiB/s, 11575 I/O per s, 667ms latency on last I/O request
Am I facing a throttle or bottleneck through VMWare or are there a bigger issue? How do I go about solving this issue as I am getting acceptable speeds on the CentOS machine and Windows machine with direct access to the SAN.
io centos vmware sysadmin iscsi
asked Nov 23 '18 at 9:43
Fifty50Fifty50
41
add a comment |
I am currently debugging guests of a host of mine that is suffering from high latency but acceptable read/write speeds on CentOS 6. This VM is connected to a VM host which has its storage on a SAN that I am reading/writing to. 7K/10K RPM HDDs.
Here is a test output (CentOS - LSI Logic SAS):
1073741824 bytes (1.1 GB) copied, 10.4863 s, 102 MB/s
Here is another test output (Windows - same SAN, different volume, VMWare Paravirtual, updated tools):
Read IO 9MiB/s, 1152 I/O per s, 582ms latency on last I/O request
Here is yet another test ouput from a machine with direct access to SAN:
Read IO 90MiB/s, 11575 I/O per s, 667ms latency on last I/O request
Am I facing a throttle or bottleneck through VMWare or are there a bigger issue? How do I go about solving this issue as I am getting acceptable speeds on the CentOS machine and Windows machine with direct access to the SAN.
io centos vmware sysadmin iscsi
asked Nov 23 '18 at 9:43
Fifty50Fifty50
41
Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…
– Kyle Ruddy
Nov 23 '18 at 20:08
@KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?
– Fifty50
Nov 27 '18 at 10:36
add a comment |
I am currently debugging guests of a host of mine that is suffering from high latency but acceptable read/write speeds on CentOS 6. This VM is connected to a VM host which has its storage on a SAN that I am reading/writing to. 7K/10K RPM HDDs.
Here is a test output (CentOS - LSI Logic SAS):
1073741824 bytes (1.1 GB) copied, 10.4863 s, 102 MB/s
Here is another test output (Windows - same SAN, different volume, VMWare Paravirtual, updated tools):
Read IO 9MiB/s, 1152 I/O per s, 582ms latency on last I/O request
Here is yet another test ouput from a machine with direct access to SAN:
Read IO 90MiB/s, 11575 I/O per s, 667ms latency on last I/O request
Am I facing a throttle or bottleneck through VMWare or are there a bigger issue? How do I go about solving this issue as I am getting acceptable speeds on the CentOS machine and Windows machine with direct access to the SAN.
io centos vmware sysadmin iscsi
asked Nov 23 '18 at 9:43
Fifty50Fifty50
41
I am currently debugging guests of a host of mine that is suffering from high latency but acceptable read/write speeds on CentOS 6. This VM is connected to a VM host which has its storage on a SAN that I am reading/writing to. 7K/10K RPM HDDs.
Here is a test output (CentOS - LSI Logic SAS):
1073741824 bytes (1.1 GB) copied, 10.4863 s, 102 MB/s
Here is another test output (Windows - same SAN, different volume, VMWare Paravirtual, updated tools):
Read IO 9MiB/s, 1152 I/O per s, 582ms latency on last I/O request
Here is yet another test ouput from a machine with direct access to SAN:
Read IO 90MiB/s, 11575 I/O per s, 667ms latency on last I/O request
Am I facing a throttle or bottleneck through VMWare or are there a bigger issue? How do I go about solving this issue as I am getting acceptable speeds on the CentOS machine and Windows machine with direct access to the SAN.
io centos vmware sysadmin iscsi
io centos vmware sysadmin iscsi
asked Nov 23 '18 at 9:43
Fifty50Fifty50
41
asked Nov 23 '18 at 9:43
Fifty50Fifty50
41
asked Nov 23 '18 at 9:43
Fifty50Fifty50
41
asked Nov 23 '18 at 9:43
Fifty50Fifty50
41
asked Nov 23 '18 at 9:43
Fifty50Fifty50
41
41
Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…
– Kyle Ruddy
Nov 23 '18 at 20:08
@KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?
– Fifty50
Nov 27 '18 at 10:36
add a comment |
Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…
– Kyle Ruddy
Nov 23 '18 at 20:08
@KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?
– Fifty50
Nov 27 '18 at 10:36
Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…
– Kyle Ruddy
Nov 23 '18 at 20:08
Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…
– Kyle Ruddy
Nov 23 '18 at 20:08
@KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?
– Fifty50
Nov 27 '18 at 10:36
@KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?
– Fifty50
Nov 27 '18 at 10:36
add a comment |
0
active
oldest
votes
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53444128%2fslow-i-o-on-vm-hosts%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53444128%2fslow-i-o-on-vm-hosts%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…
– Kyle Ruddy
Nov 23 '18 at 20:08
@KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?
– Fifty50
Nov 27 '18 at 10:36