Message sent over socket missing the n

I am generating a protocol for a tcpip socket between python and matlab. While trying to setup some sort of a protocol, I ran into a problem. It has to do with this set of code below

FPath = Path('c:/test/dogg.jpg')     

HASH = Commons.get_file_md5_hash((FPath))



msg = ('IDINFO'+FPath.name+'HASH'+ HASH+'n')

generates the message

IDINFOdogg.jpgHASH7ad1a930dab3c099939b66267b5c57f8

I have in the message: IDINFO which will tell the server the name of the file and HASH which will tell the file details.

After this I open up the file using

f = open(FPath,"rb")

chunk = f.read(1020)

and build a package with the tag DATA in front

msg = b`DATA` + chunk + b'n'

Problem is that the b'n' is not the same as in the first message. as Matlab cannot read the delimiter and won't continue grabbing data chunks.

Matlab code for Below reference. This isn't the entire object just the part that is potentially causing trouble.
To setup a callback.

       set(gh.tcpipServer, 'BytesAvailableFcnMode','Terminator');

       set(gh.tcpipServer, 'BytesAvailableFcn', @(h,e)gh.Serverpull(h,e));

The Function for looking at the bytes

    function Serverpull(gh,h,e)

        gh.msg = fread(gh.tcpipServer,gh.tcpipServer.BytesAvailable);

        gh.msgdecode = char(transpose(gh.msg));



        if strfind(gh.msgdecode,'IDINFO')

            Hst = strfind(gh.msgdecode,'HASH');

            gh.Fname = gh.msgdecode(7:Hst-1);

            gh.HASH = gh.msgdecode(Hst+4:end);

            fwrite(gh.tcpipServer,'GoodToGo');



            gh.PrepareforDataAq()

        elseif strfind(gh.msgdecode,'DATA')

            fwrite(gh.fileID,gh.msg(5:end),'double');

        elseif strfind(gh.msgdecode,'EOF')

            fclose(gh.fileID);

            display('File Transfer Complete')

        end

    end

    function PrepareforDataAq(gh)

        path = fullfile('c:temp',gh.Fname);

        gh.fileID = fopen(path,'w');



    end

For the TLDR,

How to make the string 'n' the same as b n when building a tcp message from binary instead of strings before encoding?

asked Nov 21 '18 at 8:51

Hojo.Timberwolf

398214

1

Have you tried replacing b'n' by a plain 10? BTW, you can't use a callback triggered by terminator detection if you are passing arbitrary binary data, unless you add in the protocol a specific mechanism to escape binary values equal to 10 (ASCII value for n) or make sure there is not any, e.g. convert to hex representation or base64. Otherwise, tcpip object will detect any of these bytes as an end of message. Otherwise, udp has a DatagramReceivedFcn callback better suited for that (triggered on datagram reception rather than number of bytes or terminator).

– Brice
Nov 21 '18 at 9:10

IC, Thanks for the info. I shall set it to detect by Bytes then. I can send Byte size in the initial message and have matlab count down. Hopefully that way it won't hang on the last few bytes.

– Hojo.Timberwolf
Nov 21 '18 at 9:16

Well that failed. apperently matlab won't let you set the BytesAvailableFcnMode while the connection is open. Back to the drawing board.

– Hojo.Timberwolf
Nov 21 '18 at 9:47

1

You always have the option of reading all data coming, store it in an internal buffer then process the buffer. Like, each time n bytes are received, read the object buffer and concatenate it into a Buffer variable, then parse messages in the buffer variable (which may now be mixed text/binary: if you read IDINFO look for a n terminator, if you read DATA0123 take another 123 bytes). You still have to deal with the case where the transferred file contains some of the reserved keywords (like, try to transfer a txt file documenting the protocol messages)

– Brice
Nov 21 '18 at 9:57

add a comment |

I am generating a protocol for a tcpip socket between python and matlab. While trying to setup some sort of a protocol, I ran into a problem. It has to do with this set of code below

FPath = Path('c:/test/dogg.jpg')     

HASH = Commons.get_file_md5_hash((FPath))



msg = ('IDINFO'+FPath.name+'HASH'+ HASH+'n')

generates the message

IDINFOdogg.jpgHASH7ad1a930dab3c099939b66267b5c57f8

I have in the message: IDINFO which will tell the server the name of the file and HASH which will tell the file details.

After this I open up the file using

f = open(FPath,"rb")

chunk = f.read(1020)

and build a package with the tag DATA in front

msg = b`DATA` + chunk + b'n'

Problem is that the b'n' is not the same as in the first message. as Matlab cannot read the delimiter and won't continue grabbing data chunks.

Matlab code for Below reference. This isn't the entire object just the part that is potentially causing trouble.
To setup a callback.

       set(gh.tcpipServer, 'BytesAvailableFcnMode','Terminator');

       set(gh.tcpipServer, 'BytesAvailableFcn', @(h,e)gh.Serverpull(h,e));

The Function for looking at the bytes

    function Serverpull(gh,h,e)

        gh.msg = fread(gh.tcpipServer,gh.tcpipServer.BytesAvailable);

        gh.msgdecode = char(transpose(gh.msg));



        if strfind(gh.msgdecode,'IDINFO')

            Hst = strfind(gh.msgdecode,'HASH');

            gh.Fname = gh.msgdecode(7:Hst-1);

            gh.HASH = gh.msgdecode(Hst+4:end);

            fwrite(gh.tcpipServer,'GoodToGo');



            gh.PrepareforDataAq()

        elseif strfind(gh.msgdecode,'DATA')

            fwrite(gh.fileID,gh.msg(5:end),'double');

        elseif strfind(gh.msgdecode,'EOF')

            fclose(gh.fileID);

            display('File Transfer Complete')

        end

    end

    function PrepareforDataAq(gh)

        path = fullfile('c:temp',gh.Fname);

        gh.fileID = fopen(path,'w');



    end

For the TLDR,

How to make the string 'n' the same as b n when building a tcp message from binary instead of strings before encoding?

asked Nov 21 '18 at 8:51

Hojo.Timberwolf

398214

1

Have you tried replacing b'n' by a plain 10? BTW, you can't use a callback triggered by terminator detection if you are passing arbitrary binary data, unless you add in the protocol a specific mechanism to escape binary values equal to 10 (ASCII value for n) or make sure there is not any, e.g. convert to hex representation or base64. Otherwise, tcpip object will detect any of these bytes as an end of message. Otherwise, udp has a DatagramReceivedFcn callback better suited for that (triggered on datagram reception rather than number of bytes or terminator).

– Brice
Nov 21 '18 at 9:10

IC, Thanks for the info. I shall set it to detect by Bytes then. I can send Byte size in the initial message and have matlab count down. Hopefully that way it won't hang on the last few bytes.

– Hojo.Timberwolf
Nov 21 '18 at 9:16

Well that failed. apperently matlab won't let you set the BytesAvailableFcnMode while the connection is open. Back to the drawing board.

– Hojo.Timberwolf
Nov 21 '18 at 9:47

1

You always have the option of reading all data coming, store it in an internal buffer then process the buffer. Like, each time n bytes are received, read the object buffer and concatenate it into a Buffer variable, then parse messages in the buffer variable (which may now be mixed text/binary: if you read IDINFO look for a n terminator, if you read DATA0123 take another 123 bytes). You still have to deal with the case where the transferred file contains some of the reserved keywords (like, try to transfer a txt file documenting the protocol messages)

– Brice
Nov 21 '18 at 9:57

add a comment |

I am generating a protocol for a tcpip socket between python and matlab. While trying to setup some sort of a protocol, I ran into a problem. It has to do with this set of code below

FPath = Path('c:/test/dogg.jpg')     

HASH = Commons.get_file_md5_hash((FPath))



msg = ('IDINFO'+FPath.name+'HASH'+ HASH+'n')

generates the message

IDINFOdogg.jpgHASH7ad1a930dab3c099939b66267b5c57f8

I have in the message: IDINFO which will tell the server the name of the file and HASH which will tell the file details.

After this I open up the file using

f = open(FPath,"rb")

chunk = f.read(1020)

and build a package with the tag DATA in front

msg = b`DATA` + chunk + b'n'

Problem is that the b'n' is not the same as in the first message. as Matlab cannot read the delimiter and won't continue grabbing data chunks.

Matlab code for Below reference. This isn't the entire object just the part that is potentially causing trouble.
To setup a callback.

       set(gh.tcpipServer, 'BytesAvailableFcnMode','Terminator');

       set(gh.tcpipServer, 'BytesAvailableFcn', @(h,e)gh.Serverpull(h,e));

The Function for looking at the bytes

    function Serverpull(gh,h,e)

        gh.msg = fread(gh.tcpipServer,gh.tcpipServer.BytesAvailable);

        gh.msgdecode = char(transpose(gh.msg));



        if strfind(gh.msgdecode,'IDINFO')

            Hst = strfind(gh.msgdecode,'HASH');

            gh.Fname = gh.msgdecode(7:Hst-1);

            gh.HASH = gh.msgdecode(Hst+4:end);

            fwrite(gh.tcpipServer,'GoodToGo');



            gh.PrepareforDataAq()

        elseif strfind(gh.msgdecode,'DATA')

            fwrite(gh.fileID,gh.msg(5:end),'double');

        elseif strfind(gh.msgdecode,'EOF')

            fclose(gh.fileID);

            display('File Transfer Complete')

        end

    end

    function PrepareforDataAq(gh)

        path = fullfile('c:temp',gh.Fname);

        gh.fileID = fopen(path,'w');



    end

For the TLDR,

How to make the string 'n' the same as b n when building a tcp message from binary instead of strings before encoding?

asked Nov 21 '18 at 8:51

Hojo.Timberwolf

398214

I am generating a protocol for a tcpip socket between python and matlab. While trying to setup some sort of a protocol, I ran into a problem. It has to do with this set of code below

FPath = Path('c:/test/dogg.jpg')     

HASH = Commons.get_file_md5_hash((FPath))



msg = ('IDINFO'+FPath.name+'HASH'+ HASH+'n')

generates the message

IDINFOdogg.jpgHASH7ad1a930dab3c099939b66267b5c57f8

I have in the message: IDINFO which will tell the server the name of the file and HASH which will tell the file details.

After this I open up the file using

f = open(FPath,"rb")

chunk = f.read(1020)

and build a package with the tag DATA in front

msg = b`DATA` + chunk + b'n'

Problem is that the b'n' is not the same as in the first message. as Matlab cannot read the delimiter and won't continue grabbing data chunks.

Matlab code for Below reference. This isn't the entire object just the part that is potentially causing trouble.
To setup a callback.

       set(gh.tcpipServer, 'BytesAvailableFcnMode','Terminator');

       set(gh.tcpipServer, 'BytesAvailableFcn', @(h,e)gh.Serverpull(h,e));

The Function for looking at the bytes

    function Serverpull(gh,h,e)

        gh.msg = fread(gh.tcpipServer,gh.tcpipServer.BytesAvailable);

        gh.msgdecode = char(transpose(gh.msg));



        if strfind(gh.msgdecode,'IDINFO')

            Hst = strfind(gh.msgdecode,'HASH');

            gh.Fname = gh.msgdecode(7:Hst-1);

            gh.HASH = gh.msgdecode(Hst+4:end);

            fwrite(gh.tcpipServer,'GoodToGo');



            gh.PrepareforDataAq()

        elseif strfind(gh.msgdecode,'DATA')

            fwrite(gh.fileID,gh.msg(5:end),'double');

        elseif strfind(gh.msgdecode,'EOF')

            fclose(gh.fileID);

            display('File Transfer Complete')

        end

    end

    function PrepareforDataAq(gh)

        path = fullfile('c:temp',gh.Fname);

        gh.fileID = fopen(path,'w');



    end

For the TLDR,

How to make the string 'n' the same as b n when building a tcp message from binary instead of strings before encoding?

python matlab sockets

asked Nov 21 '18 at 8:51

Hojo.Timberwolf

398214

asked Nov 21 '18 at 8:51

Hojo.Timberwolf

398214

asked Nov 21 '18 at 8:51

Hojo.Timberwolf

398214

asked Nov 21 '18 at 8:51

Hojo.Timberwolf

398214

asked Nov 21 '18 at 8:51

Hojo.Timberwolf

398214

1

Have you tried replacing b'n' by a plain 10? BTW, you can't use a callback triggered by terminator detection if you are passing arbitrary binary data, unless you add in the protocol a specific mechanism to escape binary values equal to 10 (ASCII value for n) or make sure there is not any, e.g. convert to hex representation or base64. Otherwise, tcpip object will detect any of these bytes as an end of message. Otherwise, udp has a DatagramReceivedFcn callback better suited for that (triggered on datagram reception rather than number of bytes or terminator).

– Brice
Nov 21 '18 at 9:10

IC, Thanks for the info. I shall set it to detect by Bytes then. I can send Byte size in the initial message and have matlab count down. Hopefully that way it won't hang on the last few bytes.

– Hojo.Timberwolf
Nov 21 '18 at 9:16

Well that failed. apperently matlab won't let you set the BytesAvailableFcnMode while the connection is open. Back to the drawing board.

– Hojo.Timberwolf
Nov 21 '18 at 9:47

1

You always have the option of reading all data coming, store it in an internal buffer then process the buffer. Like, each time n bytes are received, read the object buffer and concatenate it into a Buffer variable, then parse messages in the buffer variable (which may now be mixed text/binary: if you read IDINFO look for a n terminator, if you read DATA0123 take another 123 bytes). You still have to deal with the case where the transferred file contains some of the reserved keywords (like, try to transfer a txt file documenting the protocol messages)

– Brice
Nov 21 '18 at 9:57

add a comment |

1

Have you tried replacing b'n' by a plain 10? BTW, you can't use a callback triggered by terminator detection if you are passing arbitrary binary data, unless you add in the protocol a specific mechanism to escape binary values equal to 10 (ASCII value for n) or make sure there is not any, e.g. convert to hex representation or base64. Otherwise, tcpip object will detect any of these bytes as an end of message. Otherwise, udp has a DatagramReceivedFcn callback better suited for that (triggered on datagram reception rather than number of bytes or terminator).

– Brice
Nov 21 '18 at 9:10

IC, Thanks for the info. I shall set it to detect by Bytes then. I can send Byte size in the initial message and have matlab count down. Hopefully that way it won't hang on the last few bytes.

– Hojo.Timberwolf
Nov 21 '18 at 9:16

Well that failed. apperently matlab won't let you set the BytesAvailableFcnMode while the connection is open. Back to the drawing board.

– Hojo.Timberwolf
Nov 21 '18 at 9:47

1

You always have the option of reading all data coming, store it in an internal buffer then process the buffer. Like, each time n bytes are received, read the object buffer and concatenate it into a Buffer variable, then parse messages in the buffer variable (which may now be mixed text/binary: if you read IDINFO look for a n terminator, if you read DATA0123 take another 123 bytes). You still have to deal with the case where the transferred file contains some of the reserved keywords (like, try to transfer a txt file documenting the protocol messages)

– Brice
Nov 21 '18 at 9:57

Have you tried replacing b'n' by a plain 10? BTW, you can't use a callback triggered by terminator detection if you are passing arbitrary binary data, unless you add in the protocol a specific mechanism to escape binary values equal to 10 (ASCII value for n) or make sure there is not any, e.g. convert to hex representation or base64. Otherwise, tcpip object will detect any of these bytes as an end of message. Otherwise, udp has a DatagramReceivedFcn callback better suited for that (triggered on datagram reception rather than number of bytes or terminator).

– Brice
Nov 21 '18 at 9:10

IC, Thanks for the info. I shall set it to detect by Bytes then. I can send Byte size in the initial message and have matlab count down. Hopefully that way it won't hang on the last few bytes.

– Hojo.Timberwolf
Nov 21 '18 at 9:16

Well that failed. apperently matlab won't let you set the BytesAvailableFcnMode while the connection is open. Back to the drawing board.

– Hojo.Timberwolf
Nov 21 '18 at 9:47

You always have the option of reading all data coming, store it in an internal buffer then process the buffer. Like, each time n bytes are received, read the object buffer and concatenate it into a Buffer variable, then parse messages in the buffer variable (which may now be mixed text/binary: if you read IDINFO look for a n terminator, if you read DATA0123 take another 123 bytes). You still have to deal with the case where the transferred file contains some of the reserved keywords (like, try to transfer a txt file documenting the protocol messages)

– Brice
Nov 21 '18 at 9:57

add a comment |

0

active

oldest

votes

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53408288%2fmessage-sent-over-socket-missing-the-n%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

0

active

oldest

votes

0

active

oldest

votes

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Argthtjtr