Bitwise operations on strings Python3.7











up vote
1
down vote

favorite
1












I have a large string of hex numbers, on which I want to perform some bitwise operations. I am trying to extract different bytes from it and then to apply bitwise OR, AND, XOR operations. Unfortunately, I don't know an easy way to do that, so every time I want to perform bitwise operations I am converting the hex into an integer. This is a simplified code.



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"

hex_data = "0x" + data[20:32]
four_bytes = hex_data[:10]
fifth_byte = "0x" + hex_data[10:12]
lshift_fourb = hex(int(four_bytes, 16) << 1)
bitwise_or_res = hex(int(lshift_fourb, 16) | int(fifth_byte, 16))


Is there an easy way to omit the constant conversion to integer and hex back and forth in order to do the same operations. I prefer to use either hex or binary since I need to extract certain bytes from the input data string and the hex(int(hex_number, 16)) seems a bit too repetitive and tedious. If I don't convert it to an integer, Python is complaining that it cannot execute the | or ^ on strings.










share|improve this question


















  • 1




    Convert the string to an integer once, then do all operations in the integer domain, and only convert back to a string once at the end.
    – mkrieger1
    Nov 19 at 10:30






  • 1




    @mkrieger1 Integer, however, doesn't support slicing, thus calculating hex_data and four_bytes would have to be calculated with modulo. Possible, but tedious as well
    – Michal Polovka
    Nov 19 at 10:33










  • but how I am supposed to extract specific bytes from an integer?
    – Georgе Stoyanov
    Nov 19 at 10:33






  • 1




    By shifting and masking.
    – mkrieger1
    Nov 19 at 10:34






  • 1




    You can also write a few helper functions to factor out the repetitive stuff.
    – mkrieger1
    Nov 19 at 10:36















up vote
1
down vote

favorite
1












I have a large string of hex numbers, on which I want to perform some bitwise operations. I am trying to extract different bytes from it and then to apply bitwise OR, AND, XOR operations. Unfortunately, I don't know an easy way to do that, so every time I want to perform bitwise operations I am converting the hex into an integer. This is a simplified code.



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"

hex_data = "0x" + data[20:32]
four_bytes = hex_data[:10]
fifth_byte = "0x" + hex_data[10:12]
lshift_fourb = hex(int(four_bytes, 16) << 1)
bitwise_or_res = hex(int(lshift_fourb, 16) | int(fifth_byte, 16))


Is there an easy way to omit the constant conversion to integer and hex back and forth in order to do the same operations. I prefer to use either hex or binary since I need to extract certain bytes from the input data string and the hex(int(hex_number, 16)) seems a bit too repetitive and tedious. If I don't convert it to an integer, Python is complaining that it cannot execute the | or ^ on strings.










share|improve this question


















  • 1




    Convert the string to an integer once, then do all operations in the integer domain, and only convert back to a string once at the end.
    – mkrieger1
    Nov 19 at 10:30






  • 1




    @mkrieger1 Integer, however, doesn't support slicing, thus calculating hex_data and four_bytes would have to be calculated with modulo. Possible, but tedious as well
    – Michal Polovka
    Nov 19 at 10:33










  • but how I am supposed to extract specific bytes from an integer?
    – Georgе Stoyanov
    Nov 19 at 10:33






  • 1




    By shifting and masking.
    – mkrieger1
    Nov 19 at 10:34






  • 1




    You can also write a few helper functions to factor out the repetitive stuff.
    – mkrieger1
    Nov 19 at 10:36













up vote
1
down vote

favorite
1









up vote
1
down vote

favorite
1






1





I have a large string of hex numbers, on which I want to perform some bitwise operations. I am trying to extract different bytes from it and then to apply bitwise OR, AND, XOR operations. Unfortunately, I don't know an easy way to do that, so every time I want to perform bitwise operations I am converting the hex into an integer. This is a simplified code.



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"

hex_data = "0x" + data[20:32]
four_bytes = hex_data[:10]
fifth_byte = "0x" + hex_data[10:12]
lshift_fourb = hex(int(four_bytes, 16) << 1)
bitwise_or_res = hex(int(lshift_fourb, 16) | int(fifth_byte, 16))


Is there an easy way to omit the constant conversion to integer and hex back and forth in order to do the same operations. I prefer to use either hex or binary since I need to extract certain bytes from the input data string and the hex(int(hex_number, 16)) seems a bit too repetitive and tedious. If I don't convert it to an integer, Python is complaining that it cannot execute the | or ^ on strings.










share|improve this question













I have a large string of hex numbers, on which I want to perform some bitwise operations. I am trying to extract different bytes from it and then to apply bitwise OR, AND, XOR operations. Unfortunately, I don't know an easy way to do that, so every time I want to perform bitwise operations I am converting the hex into an integer. This is a simplified code.



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"

hex_data = "0x" + data[20:32]
four_bytes = hex_data[:10]
fifth_byte = "0x" + hex_data[10:12]
lshift_fourb = hex(int(four_bytes, 16) << 1)
bitwise_or_res = hex(int(lshift_fourb, 16) | int(fifth_byte, 16))


Is there an easy way to omit the constant conversion to integer and hex back and forth in order to do the same operations. I prefer to use either hex or binary since I need to extract certain bytes from the input data string and the hex(int(hex_number, 16)) seems a bit too repetitive and tedious. If I don't convert it to an integer, Python is complaining that it cannot execute the | or ^ on strings.







python python-3.x bit-manipulation bitwise-operators






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 19 at 10:27









Georgе Stoyanov

7712




7712








  • 1




    Convert the string to an integer once, then do all operations in the integer domain, and only convert back to a string once at the end.
    – mkrieger1
    Nov 19 at 10:30






  • 1




    @mkrieger1 Integer, however, doesn't support slicing, thus calculating hex_data and four_bytes would have to be calculated with modulo. Possible, but tedious as well
    – Michal Polovka
    Nov 19 at 10:33










  • but how I am supposed to extract specific bytes from an integer?
    – Georgе Stoyanov
    Nov 19 at 10:33






  • 1




    By shifting and masking.
    – mkrieger1
    Nov 19 at 10:34






  • 1




    You can also write a few helper functions to factor out the repetitive stuff.
    – mkrieger1
    Nov 19 at 10:36














  • 1




    Convert the string to an integer once, then do all operations in the integer domain, and only convert back to a string once at the end.
    – mkrieger1
    Nov 19 at 10:30






  • 1




    @mkrieger1 Integer, however, doesn't support slicing, thus calculating hex_data and four_bytes would have to be calculated with modulo. Possible, but tedious as well
    – Michal Polovka
    Nov 19 at 10:33










  • but how I am supposed to extract specific bytes from an integer?
    – Georgе Stoyanov
    Nov 19 at 10:33






  • 1




    By shifting and masking.
    – mkrieger1
    Nov 19 at 10:34






  • 1




    You can also write a few helper functions to factor out the repetitive stuff.
    – mkrieger1
    Nov 19 at 10:36








1




1




Convert the string to an integer once, then do all operations in the integer domain, and only convert back to a string once at the end.
– mkrieger1
Nov 19 at 10:30




Convert the string to an integer once, then do all operations in the integer domain, and only convert back to a string once at the end.
– mkrieger1
Nov 19 at 10:30




1




1




@mkrieger1 Integer, however, doesn't support slicing, thus calculating hex_data and four_bytes would have to be calculated with modulo. Possible, but tedious as well
– Michal Polovka
Nov 19 at 10:33




@mkrieger1 Integer, however, doesn't support slicing, thus calculating hex_data and four_bytes would have to be calculated with modulo. Possible, but tedious as well
– Michal Polovka
Nov 19 at 10:33












but how I am supposed to extract specific bytes from an integer?
– Georgе Stoyanov
Nov 19 at 10:33




but how I am supposed to extract specific bytes from an integer?
– Georgе Stoyanov
Nov 19 at 10:33




1




1




By shifting and masking.
– mkrieger1
Nov 19 at 10:34




By shifting and masking.
– mkrieger1
Nov 19 at 10:34




1




1




You can also write a few helper functions to factor out the repetitive stuff.
– mkrieger1
Nov 19 at 10:36




You can also write a few helper functions to factor out the repetitive stuff.
– mkrieger1
Nov 19 at 10:36












2 Answers
2






active

oldest

votes

















up vote
2
down vote



accepted










How about this:



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"
size = (len(data)-2)//2
data_bytes = int(data,16).to_bytes(size,byteorder='big')


Now you can do this:



data_bytes[4] & data_bytes[5]





share|improve this answer























  • this seems to work, but I don't know why, when I print data_bytes it gives me b'+Cxf5xe3~m' I am expecting something like b'x2bx43xf5xe3x7ex6d and also how I can extract the first 4 bytes from data_bytes. I am also not able to perform bitwise operations on bytes and integers, so I need either to convert the bytes to integers or convert the integers to bytes.
    – Georgе Stoyanov
    Nov 19 at 11:26






  • 1




    I have understood why the format of the data_bytes, I need to print them with .hex() at the end, otherwise, it is trying to match the ASCII codes.
    – Georgе Stoyanov
    Nov 19 at 11:34










  • I mean when I try to execute data_byte[3] << bytes(1) it gives me an error that the left shift operation is not supported for 'bytes' objects.
    – Georgе Stoyanov
    Nov 19 at 11:41










  • As a workaround I need to convert the bytes to integers using int.from_bytes(data_bytes, byteorder="big") in order to perform the bitwise left or right shift.
    – Georgе Stoyanov
    Nov 19 at 11:47










  • at least on my machine, printing data_bytes produces the desired result. Note that I've edited the code in the answer.
    – Yakov Dan
    Nov 19 at 13:04


















up vote
2
down vote













As you mentioned, the problem is that strings of hex digits are more readable and good for slicing and at least some shifting operations, but don't support bitwise operations. On the other hand, integers support bitwise operations but no slicing. The closest to what you want may be to create a custom class implementing both features (doing conversion whenever needed). This won't save you from implementing (and executing) the more complex code, but the rest of your application may be more readable because the conversions are "hidden".






share|improve this answer





















  • exactly my point. Hex is perfect for slicing but then I need to convert it all the time in order to execute bitwise operations.
    – Georgе Stoyanov
    Nov 19 at 11:28











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53372602%2fbitwise-operations-on-strings-python3-7%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
2
down vote



accepted










How about this:



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"
size = (len(data)-2)//2
data_bytes = int(data,16).to_bytes(size,byteorder='big')


Now you can do this:



data_bytes[4] & data_bytes[5]





share|improve this answer























  • this seems to work, but I don't know why, when I print data_bytes it gives me b'+Cxf5xe3~m' I am expecting something like b'x2bx43xf5xe3x7ex6d and also how I can extract the first 4 bytes from data_bytes. I am also not able to perform bitwise operations on bytes and integers, so I need either to convert the bytes to integers or convert the integers to bytes.
    – Georgе Stoyanov
    Nov 19 at 11:26






  • 1




    I have understood why the format of the data_bytes, I need to print them with .hex() at the end, otherwise, it is trying to match the ASCII codes.
    – Georgе Stoyanov
    Nov 19 at 11:34










  • I mean when I try to execute data_byte[3] << bytes(1) it gives me an error that the left shift operation is not supported for 'bytes' objects.
    – Georgе Stoyanov
    Nov 19 at 11:41










  • As a workaround I need to convert the bytes to integers using int.from_bytes(data_bytes, byteorder="big") in order to perform the bitwise left or right shift.
    – Georgе Stoyanov
    Nov 19 at 11:47










  • at least on my machine, printing data_bytes produces the desired result. Note that I've edited the code in the answer.
    – Yakov Dan
    Nov 19 at 13:04















up vote
2
down vote



accepted










How about this:



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"
size = (len(data)-2)//2
data_bytes = int(data,16).to_bytes(size,byteorder='big')


Now you can do this:



data_bytes[4] & data_bytes[5]





share|improve this answer























  • this seems to work, but I don't know why, when I print data_bytes it gives me b'+Cxf5xe3~m' I am expecting something like b'x2bx43xf5xe3x7ex6d and also how I can extract the first 4 bytes from data_bytes. I am also not able to perform bitwise operations on bytes and integers, so I need either to convert the bytes to integers or convert the integers to bytes.
    – Georgе Stoyanov
    Nov 19 at 11:26






  • 1




    I have understood why the format of the data_bytes, I need to print them with .hex() at the end, otherwise, it is trying to match the ASCII codes.
    – Georgе Stoyanov
    Nov 19 at 11:34










  • I mean when I try to execute data_byte[3] << bytes(1) it gives me an error that the left shift operation is not supported for 'bytes' objects.
    – Georgе Stoyanov
    Nov 19 at 11:41










  • As a workaround I need to convert the bytes to integers using int.from_bytes(data_bytes, byteorder="big") in order to perform the bitwise left or right shift.
    – Georgе Stoyanov
    Nov 19 at 11:47










  • at least on my machine, printing data_bytes produces the desired result. Note that I've edited the code in the answer.
    – Yakov Dan
    Nov 19 at 13:04













up vote
2
down vote



accepted







up vote
2
down vote



accepted






How about this:



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"
size = (len(data)-2)//2
data_bytes = int(data,16).to_bytes(size,byteorder='big')


Now you can do this:



data_bytes[4] & data_bytes[5]





share|improve this answer














How about this:



data = "0x4700ff1de05ca7686c2b43f5e37e6dafd388761c36900ab37"
size = (len(data)-2)//2
data_bytes = int(data,16).to_bytes(size,byteorder='big')


Now you can do this:



data_bytes[4] & data_bytes[5]






share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 19 at 13:02

























answered Nov 19 at 10:57









Yakov Dan

429211




429211












  • this seems to work, but I don't know why, when I print data_bytes it gives me b'+Cxf5xe3~m' I am expecting something like b'x2bx43xf5xe3x7ex6d and also how I can extract the first 4 bytes from data_bytes. I am also not able to perform bitwise operations on bytes and integers, so I need either to convert the bytes to integers or convert the integers to bytes.
    – Georgе Stoyanov
    Nov 19 at 11:26






  • 1




    I have understood why the format of the data_bytes, I need to print them with .hex() at the end, otherwise, it is trying to match the ASCII codes.
    – Georgе Stoyanov
    Nov 19 at 11:34










  • I mean when I try to execute data_byte[3] << bytes(1) it gives me an error that the left shift operation is not supported for 'bytes' objects.
    – Georgе Stoyanov
    Nov 19 at 11:41










  • As a workaround I need to convert the bytes to integers using int.from_bytes(data_bytes, byteorder="big") in order to perform the bitwise left or right shift.
    – Georgе Stoyanov
    Nov 19 at 11:47










  • at least on my machine, printing data_bytes produces the desired result. Note that I've edited the code in the answer.
    – Yakov Dan
    Nov 19 at 13:04


















  • this seems to work, but I don't know why, when I print data_bytes it gives me b'+Cxf5xe3~m' I am expecting something like b'x2bx43xf5xe3x7ex6d and also how I can extract the first 4 bytes from data_bytes. I am also not able to perform bitwise operations on bytes and integers, so I need either to convert the bytes to integers or convert the integers to bytes.
    – Georgе Stoyanov
    Nov 19 at 11:26






  • 1




    I have understood why the format of the data_bytes, I need to print them with .hex() at the end, otherwise, it is trying to match the ASCII codes.
    – Georgе Stoyanov
    Nov 19 at 11:34










  • I mean when I try to execute data_byte[3] << bytes(1) it gives me an error that the left shift operation is not supported for 'bytes' objects.
    – Georgе Stoyanov
    Nov 19 at 11:41










  • As a workaround I need to convert the bytes to integers using int.from_bytes(data_bytes, byteorder="big") in order to perform the bitwise left or right shift.
    – Georgе Stoyanov
    Nov 19 at 11:47










  • at least on my machine, printing data_bytes produces the desired result. Note that I've edited the code in the answer.
    – Yakov Dan
    Nov 19 at 13:04
















this seems to work, but I don't know why, when I print data_bytes it gives me b'+Cxf5xe3~m' I am expecting something like b'x2bx43xf5xe3x7ex6d and also how I can extract the first 4 bytes from data_bytes. I am also not able to perform bitwise operations on bytes and integers, so I need either to convert the bytes to integers or convert the integers to bytes.
– Georgе Stoyanov
Nov 19 at 11:26




this seems to work, but I don't know why, when I print data_bytes it gives me b'+Cxf5xe3~m' I am expecting something like b'x2bx43xf5xe3x7ex6d and also how I can extract the first 4 bytes from data_bytes. I am also not able to perform bitwise operations on bytes and integers, so I need either to convert the bytes to integers or convert the integers to bytes.
– Georgе Stoyanov
Nov 19 at 11:26




1




1




I have understood why the format of the data_bytes, I need to print them with .hex() at the end, otherwise, it is trying to match the ASCII codes.
– Georgе Stoyanov
Nov 19 at 11:34




I have understood why the format of the data_bytes, I need to print them with .hex() at the end, otherwise, it is trying to match the ASCII codes.
– Georgе Stoyanov
Nov 19 at 11:34












I mean when I try to execute data_byte[3] << bytes(1) it gives me an error that the left shift operation is not supported for 'bytes' objects.
– Georgе Stoyanov
Nov 19 at 11:41




I mean when I try to execute data_byte[3] << bytes(1) it gives me an error that the left shift operation is not supported for 'bytes' objects.
– Georgе Stoyanov
Nov 19 at 11:41












As a workaround I need to convert the bytes to integers using int.from_bytes(data_bytes, byteorder="big") in order to perform the bitwise left or right shift.
– Georgе Stoyanov
Nov 19 at 11:47




As a workaround I need to convert the bytes to integers using int.from_bytes(data_bytes, byteorder="big") in order to perform the bitwise left or right shift.
– Georgе Stoyanov
Nov 19 at 11:47












at least on my machine, printing data_bytes produces the desired result. Note that I've edited the code in the answer.
– Yakov Dan
Nov 19 at 13:04




at least on my machine, printing data_bytes produces the desired result. Note that I've edited the code in the answer.
– Yakov Dan
Nov 19 at 13:04












up vote
2
down vote













As you mentioned, the problem is that strings of hex digits are more readable and good for slicing and at least some shifting operations, but don't support bitwise operations. On the other hand, integers support bitwise operations but no slicing. The closest to what you want may be to create a custom class implementing both features (doing conversion whenever needed). This won't save you from implementing (and executing) the more complex code, but the rest of your application may be more readable because the conversions are "hidden".






share|improve this answer





















  • exactly my point. Hex is perfect for slicing but then I need to convert it all the time in order to execute bitwise operations.
    – Georgе Stoyanov
    Nov 19 at 11:28















up vote
2
down vote













As you mentioned, the problem is that strings of hex digits are more readable and good for slicing and at least some shifting operations, but don't support bitwise operations. On the other hand, integers support bitwise operations but no slicing. The closest to what you want may be to create a custom class implementing both features (doing conversion whenever needed). This won't save you from implementing (and executing) the more complex code, but the rest of your application may be more readable because the conversions are "hidden".






share|improve this answer





















  • exactly my point. Hex is perfect for slicing but then I need to convert it all the time in order to execute bitwise operations.
    – Georgе Stoyanov
    Nov 19 at 11:28













up vote
2
down vote










up vote
2
down vote









As you mentioned, the problem is that strings of hex digits are more readable and good for slicing and at least some shifting operations, but don't support bitwise operations. On the other hand, integers support bitwise operations but no slicing. The closest to what you want may be to create a custom class implementing both features (doing conversion whenever needed). This won't save you from implementing (and executing) the more complex code, but the rest of your application may be more readable because the conversions are "hidden".






share|improve this answer












As you mentioned, the problem is that strings of hex digits are more readable and good for slicing and at least some shifting operations, but don't support bitwise operations. On the other hand, integers support bitwise operations but no slicing. The closest to what you want may be to create a custom class implementing both features (doing conversion whenever needed). This won't save you from implementing (and executing) the more complex code, but the rest of your application may be more readable because the conversions are "hidden".







share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 19 at 11:26









jtd

986




986












  • exactly my point. Hex is perfect for slicing but then I need to convert it all the time in order to execute bitwise operations.
    – Georgе Stoyanov
    Nov 19 at 11:28


















  • exactly my point. Hex is perfect for slicing but then I need to convert it all the time in order to execute bitwise operations.
    – Georgе Stoyanov
    Nov 19 at 11:28
















exactly my point. Hex is perfect for slicing but then I need to convert it all the time in order to execute bitwise operations.
– Georgе Stoyanov
Nov 19 at 11:28




exactly my point. Hex is perfect for slicing but then I need to convert it all the time in order to execute bitwise operations.
– Georgе Stoyanov
Nov 19 at 11:28


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53372602%2fbitwise-operations-on-strings-python3-7%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

"Incorrect syntax near the keyword 'ON'. (on update cascade, on delete cascade,)

Alcedinidae

RAC Tourist Trophy