How to deal with huge amount of byte arrays
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...
So this is the receiver:
static class ReceiveLogsThread implements Runnable {
private static final String TAG = "IoTReceiveLogsThread";
Message msgRead;
ReceiveLogsThread() {
}
public void run() {
byte rbuf = new byte[4096];
while (!Thread.currentThread().isInterrupted()) {
try {
int len = mSerial.readLog(rbuf, mSerialPortLog);
if (len > 0) {
// Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
// this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
receiveLogsHandler.sendMessage(this.msgRead);
}
} catch (NullPointerException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.currentThread().interrupt();
if (!mReadFlag) {
Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
}
}
}
As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation
private static String printHex(byte bytes) {
Formatter formatter = new Formatter();
for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}
I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..
I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...
Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:
private List<byte> logs = new ArrayList<>();
Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
public void handleMessage(Message msgRW) {
super.handleMessage(msgRW);
// logMessagesList.add(msgRW.obj.toString().toUpperCase());
// String message = msgRW.obj.toString().toUpperCase();
if(shouldCollectLogs) {
byte message = (byte) msgRW.obj;
logs.add(message);
}
....
So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..
java
|
show 4 more comments
So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...
So this is the receiver:
static class ReceiveLogsThread implements Runnable {
private static final String TAG = "IoTReceiveLogsThread";
Message msgRead;
ReceiveLogsThread() {
}
public void run() {
byte rbuf = new byte[4096];
while (!Thread.currentThread().isInterrupted()) {
try {
int len = mSerial.readLog(rbuf, mSerialPortLog);
if (len > 0) {
// Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
// this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
receiveLogsHandler.sendMessage(this.msgRead);
}
} catch (NullPointerException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.currentThread().interrupt();
if (!mReadFlag) {
Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
}
}
}
As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation
private static String printHex(byte bytes) {
Formatter formatter = new Formatter();
for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}
I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..
I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...
Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:
private List<byte> logs = new ArrayList<>();
Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
public void handleMessage(Message msgRW) {
super.handleMessage(msgRW);
// logMessagesList.add(msgRW.obj.toString().toUpperCase());
// String message = msgRW.obj.toString().toUpperCase();
if(shouldCollectLogs) {
byte message = (byte) msgRW.obj;
logs.add(message);
}
....
So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..
java
Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html
– dglozano
Nov 23 '18 at 18:35
Thank you my friend, but I don't have time to implement that :)
– joe
Nov 23 '18 at 18:38
So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now
– dglozano
Nov 23 '18 at 19:08
The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...
– joe
Nov 23 '18 at 19:11
Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end
– dglozano
Nov 23 '18 at 19:18
|
show 4 more comments
So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...
So this is the receiver:
static class ReceiveLogsThread implements Runnable {
private static final String TAG = "IoTReceiveLogsThread";
Message msgRead;
ReceiveLogsThread() {
}
public void run() {
byte rbuf = new byte[4096];
while (!Thread.currentThread().isInterrupted()) {
try {
int len = mSerial.readLog(rbuf, mSerialPortLog);
if (len > 0) {
// Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
// this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
receiveLogsHandler.sendMessage(this.msgRead);
}
} catch (NullPointerException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.currentThread().interrupt();
if (!mReadFlag) {
Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
}
}
}
As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation
private static String printHex(byte bytes) {
Formatter formatter = new Formatter();
for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}
I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..
I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...
Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:
private List<byte> logs = new ArrayList<>();
Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
public void handleMessage(Message msgRW) {
super.handleMessage(msgRW);
// logMessagesList.add(msgRW.obj.toString().toUpperCase());
// String message = msgRW.obj.toString().toUpperCase();
if(shouldCollectLogs) {
byte message = (byte) msgRW.obj;
logs.add(message);
}
....
So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..
java
So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...
So this is the receiver:
static class ReceiveLogsThread implements Runnable {
private static final String TAG = "IoTReceiveLogsThread";
Message msgRead;
ReceiveLogsThread() {
}
public void run() {
byte rbuf = new byte[4096];
while (!Thread.currentThread().isInterrupted()) {
try {
int len = mSerial.readLog(rbuf, mSerialPortLog);
if (len > 0) {
// Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
// this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
receiveLogsHandler.sendMessage(this.msgRead);
}
} catch (NullPointerException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.currentThread().interrupt();
if (!mReadFlag) {
Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
}
}
}
As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation
private static String printHex(byte bytes) {
Formatter formatter = new Formatter();
for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}
I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..
I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...
Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:
private List<byte> logs = new ArrayList<>();
Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
public void handleMessage(Message msgRW) {
super.handleMessage(msgRW);
// logMessagesList.add(msgRW.obj.toString().toUpperCase());
// String message = msgRW.obj.toString().toUpperCase();
if(shouldCollectLogs) {
byte message = (byte) msgRW.obj;
logs.add(message);
}
....
So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..
java
java
asked Nov 23 '18 at 18:07
joejoe
1832824
1832824
Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html
– dglozano
Nov 23 '18 at 18:35
Thank you my friend, but I don't have time to implement that :)
– joe
Nov 23 '18 at 18:38
So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now
– dglozano
Nov 23 '18 at 19:08
The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...
– joe
Nov 23 '18 at 19:11
Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end
– dglozano
Nov 23 '18 at 19:18
|
show 4 more comments
Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html
– dglozano
Nov 23 '18 at 18:35
Thank you my friend, but I don't have time to implement that :)
– joe
Nov 23 '18 at 18:38
So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now
– dglozano
Nov 23 '18 at 19:08
The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...
– joe
Nov 23 '18 at 19:11
Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end
– dglozano
Nov 23 '18 at 19:18
Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html
– dglozano
Nov 23 '18 at 18:35
Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html
– dglozano
Nov 23 '18 at 18:35
Thank you my friend, but I don't have time to implement that :)
– joe
Nov 23 '18 at 18:38
Thank you my friend, but I don't have time to implement that :)
– joe
Nov 23 '18 at 18:38
So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now
– dglozano
Nov 23 '18 at 19:08
So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now
– dglozano
Nov 23 '18 at 19:08
The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...
– joe
Nov 23 '18 at 19:11
The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...
– joe
Nov 23 '18 at 19:11
Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end
– dglozano
Nov 23 '18 at 19:18
Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end
– dglozano
Nov 23 '18 at 19:18
|
show 4 more comments
2 Answers
2
active
oldest
votes
Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.
private final static char hexArray = "0123456789ABCDEF".toCharArray();
public static String printToHex(byte bytes, int len) {
char hexChars = new char[len * 2];
for ( int j = 0; j < len; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one
– joe
Nov 23 '18 at 20:20
@joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.
– lionscribe
Nov 23 '18 at 20:56
It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"
– joe
Nov 24 '18 at 2:20
add a comment |
Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.
Alternative 1: Collect to List of String
List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.toList());// Collect the streams into one final list
Alternative 2: Collect to one String
String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.joining()); // Concat the streams into one final String
You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53451247%2fhow-to-deal-with-huge-amount-of-byte-arrays%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.
private final static char hexArray = "0123456789ABCDEF".toCharArray();
public static String printToHex(byte bytes, int len) {
char hexChars = new char[len * 2];
for ( int j = 0; j < len; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one
– joe
Nov 23 '18 at 20:20
@joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.
– lionscribe
Nov 23 '18 at 20:56
It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"
– joe
Nov 24 '18 at 2:20
add a comment |
Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.
private final static char hexArray = "0123456789ABCDEF".toCharArray();
public static String printToHex(byte bytes, int len) {
char hexChars = new char[len * 2];
for ( int j = 0; j < len; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one
– joe
Nov 23 '18 at 20:20
@joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.
– lionscribe
Nov 23 '18 at 20:56
It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"
– joe
Nov 24 '18 at 2:20
add a comment |
Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.
private final static char hexArray = "0123456789ABCDEF".toCharArray();
public static String printToHex(byte bytes, int len) {
char hexChars = new char[len * 2];
for ( int j = 0; j < len; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.
private final static char hexArray = "0123456789ABCDEF".toCharArray();
public static String printToHex(byte bytes, int len) {
char hexChars = new char[len * 2];
for ( int j = 0; j < len; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
answered Nov 23 '18 at 19:47
lionscribelionscribe
2,1291714
2,1291714
Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one
– joe
Nov 23 '18 at 20:20
@joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.
– lionscribe
Nov 23 '18 at 20:56
It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"
– joe
Nov 24 '18 at 2:20
add a comment |
Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one
– joe
Nov 23 '18 at 20:20
@joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.
– lionscribe
Nov 23 '18 at 20:56
It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"
– joe
Nov 24 '18 at 2:20
Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one
– joe
Nov 23 '18 at 20:20
Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one
– joe
Nov 23 '18 at 20:20
@joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.
– lionscribe
Nov 23 '18 at 20:56
@joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.
– lionscribe
Nov 23 '18 at 20:56
It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"
– joe
Nov 24 '18 at 2:20
It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"
– joe
Nov 24 '18 at 2:20
add a comment |
Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.
Alternative 1: Collect to List of String
List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.toList());// Collect the streams into one final list
Alternative 2: Collect to one String
String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.joining()); // Concat the streams into one final String
You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.
add a comment |
Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.
Alternative 1: Collect to List of String
List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.toList());// Collect the streams into one final list
Alternative 2: Collect to one String
String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.joining()); // Concat the streams into one final String
You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.
add a comment |
Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.
Alternative 1: Collect to List of String
List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.toList());// Collect the streams into one final list
Alternative 2: Collect to one String
String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.joining()); // Concat the streams into one final String
You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.
Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.
Alternative 1: Collect to List of String
List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.toList());// Collect the streams into one final list
Alternative 2: Collect to one String
String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.joining()); // Concat the streams into one final String
You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.
answered Nov 23 '18 at 19:31
dglozanodglozano
1,116221
1,116221
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53451247%2fhow-to-deal-with-huge-amount-of-byte-arrays%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html
– dglozano
Nov 23 '18 at 18:35
Thank you my friend, but I don't have time to implement that :)
– joe
Nov 23 '18 at 18:38
So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now
– dglozano
Nov 23 '18 at 19:08
The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...
– joe
Nov 23 '18 at 19:11
Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end
– dglozano
Nov 23 '18 at 19:18