Slow I/O on VM hosts












0















I am currently debugging guests of a host of mine that is suffering from high latency but acceptable read/write speeds on CentOS 6. This VM is connected to a VM host which has its storage on a SAN that I am reading/writing to. 7K/10K RPM HDDs.



Here is a test output (CentOS - LSI Logic SAS):



1073741824 bytes (1.1 GB) copied, 10.4863 s, 102 MB/s


Here is another test output (Windows - same SAN, different volume, VMWare Paravirtual, updated tools):



Read IO 9MiB/s, 1152 I/O per s, 582ms latency on last I/O request


Here is yet another test ouput from a machine with direct access to SAN:



Read IO 90MiB/s, 11575 I/O per s, 667ms latency on last I/O request


Am I facing a throttle or bottleneck through VMWare or are there a bigger issue? How do I go about solving this issue as I am getting acceptable speeds on the CentOS machine and Windows machine with direct access to the SAN.










share|improve this question























  • Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…

    – Kyle Ruddy
    Nov 23 '18 at 20:08











  • @KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?

    – Fifty50
    Nov 27 '18 at 10:36
















0















I am currently debugging guests of a host of mine that is suffering from high latency but acceptable read/write speeds on CentOS 6. This VM is connected to a VM host which has its storage on a SAN that I am reading/writing to. 7K/10K RPM HDDs.



Here is a test output (CentOS - LSI Logic SAS):



1073741824 bytes (1.1 GB) copied, 10.4863 s, 102 MB/s


Here is another test output (Windows - same SAN, different volume, VMWare Paravirtual, updated tools):



Read IO 9MiB/s, 1152 I/O per s, 582ms latency on last I/O request


Here is yet another test ouput from a machine with direct access to SAN:



Read IO 90MiB/s, 11575 I/O per s, 667ms latency on last I/O request


Am I facing a throttle or bottleneck through VMWare or are there a bigger issue? How do I go about solving this issue as I am getting acceptable speeds on the CentOS machine and Windows machine with direct access to the SAN.










share|improve this question























  • Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…

    – Kyle Ruddy
    Nov 23 '18 at 20:08











  • @KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?

    – Fifty50
    Nov 27 '18 at 10:36














0












0








0








I am currently debugging guests of a host of mine that is suffering from high latency but acceptable read/write speeds on CentOS 6. This VM is connected to a VM host which has its storage on a SAN that I am reading/writing to. 7K/10K RPM HDDs.



Here is a test output (CentOS - LSI Logic SAS):



1073741824 bytes (1.1 GB) copied, 10.4863 s, 102 MB/s


Here is another test output (Windows - same SAN, different volume, VMWare Paravirtual, updated tools):



Read IO 9MiB/s, 1152 I/O per s, 582ms latency on last I/O request


Here is yet another test ouput from a machine with direct access to SAN:



Read IO 90MiB/s, 11575 I/O per s, 667ms latency on last I/O request


Am I facing a throttle or bottleneck through VMWare or are there a bigger issue? How do I go about solving this issue as I am getting acceptable speeds on the CentOS machine and Windows machine with direct access to the SAN.










share|improve this question














I am currently debugging guests of a host of mine that is suffering from high latency but acceptable read/write speeds on CentOS 6. This VM is connected to a VM host which has its storage on a SAN that I am reading/writing to. 7K/10K RPM HDDs.



Here is a test output (CentOS - LSI Logic SAS):



1073741824 bytes (1.1 GB) copied, 10.4863 s, 102 MB/s


Here is another test output (Windows - same SAN, different volume, VMWare Paravirtual, updated tools):



Read IO 9MiB/s, 1152 I/O per s, 582ms latency on last I/O request


Here is yet another test ouput from a machine with direct access to SAN:



Read IO 90MiB/s, 11575 I/O per s, 667ms latency on last I/O request


Am I facing a throttle or bottleneck through VMWare or are there a bigger issue? How do I go about solving this issue as I am getting acceptable speeds on the CentOS machine and Windows machine with direct access to the SAN.







io centos vmware sysadmin iscsi






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 23 '18 at 9:43









Fifty50Fifty50

41




41













  • Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…

    – Kyle Ruddy
    Nov 23 '18 at 20:08











  • @KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?

    – Fifty50
    Nov 27 '18 at 10:36



















  • Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…

    – Kyle Ruddy
    Nov 23 '18 at 20:08











  • @KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?

    – Fifty50
    Nov 27 '18 at 10:36

















Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…

– Kyle Ruddy
Nov 23 '18 at 20:08





Probably worth diving into ESXTOP on your ESXi host and looking into some of the latency values to try and determine the bottleneck. The following blog has some examples towards the middle around storage performance troubleshooting: virtualizationhowto.com/2018/08/…

– Kyle Ruddy
Nov 23 '18 at 20:08













@KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?

– Fifty50
Nov 27 '18 at 10:36





@KyleRuddy Thanks for the information. I used the esxtop command to determine the cause but I expected to see a different result of kernel latency or something but I ended up seeing guest latency spiking. Does this mean it is a guest issue and not a host issue --> SAN issue?

– Fifty50
Nov 27 '18 at 10:36












0






active

oldest

votes












Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53444128%2fslow-i-o-on-vm-hosts%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53444128%2fslow-i-o-on-vm-hosts%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

"Incorrect syntax near the keyword 'ON'. (on update cascade, on delete cascade,)

Alcedinidae

Origin of the phrase “under your belt”?