How to deal with huge amount of byte arrays





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}







0















So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...



So this is the receiver:



static class ReceiveLogsThread implements Runnable {
private static final String TAG = "IoTReceiveLogsThread";
Message msgRead;

ReceiveLogsThread() {
}

public void run() {
byte rbuf = new byte[4096];
while (!Thread.currentThread().isInterrupted()) {
try {
int len = mSerial.readLog(rbuf, mSerialPortLog);
if (len > 0) {
// Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
// this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
receiveLogsHandler.sendMessage(this.msgRead);
}
} catch (NullPointerException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.currentThread().interrupt();
if (!mReadFlag) {
Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
}
}
}


As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation



   private static String printHex(byte bytes) {
Formatter formatter = new Formatter();

for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}


I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..



I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...



Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:



    private List<byte> logs = new ArrayList<>();

Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
public void handleMessage(Message msgRW) {
super.handleMessage(msgRW);
// logMessagesList.add(msgRW.obj.toString().toUpperCase());
// String message = msgRW.obj.toString().toUpperCase();

if(shouldCollectLogs) {
byte message = (byte) msgRW.obj;
logs.add(message);
}

....


So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..










share|improve this question























  • Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html

    – dglozano
    Nov 23 '18 at 18:35











  • Thank you my friend, but I don't have time to implement that :)

    – joe
    Nov 23 '18 at 18:38











  • So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now

    – dglozano
    Nov 23 '18 at 19:08











  • The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...

    – joe
    Nov 23 '18 at 19:11











  • Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end

    – dglozano
    Nov 23 '18 at 19:18


















0















So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...



So this is the receiver:



static class ReceiveLogsThread implements Runnable {
private static final String TAG = "IoTReceiveLogsThread";
Message msgRead;

ReceiveLogsThread() {
}

public void run() {
byte rbuf = new byte[4096];
while (!Thread.currentThread().isInterrupted()) {
try {
int len = mSerial.readLog(rbuf, mSerialPortLog);
if (len > 0) {
// Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
// this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
receiveLogsHandler.sendMessage(this.msgRead);
}
} catch (NullPointerException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.currentThread().interrupt();
if (!mReadFlag) {
Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
}
}
}


As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation



   private static String printHex(byte bytes) {
Formatter formatter = new Formatter();

for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}


I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..



I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...



Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:



    private List<byte> logs = new ArrayList<>();

Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
public void handleMessage(Message msgRW) {
super.handleMessage(msgRW);
// logMessagesList.add(msgRW.obj.toString().toUpperCase());
// String message = msgRW.obj.toString().toUpperCase();

if(shouldCollectLogs) {
byte message = (byte) msgRW.obj;
logs.add(message);
}

....


So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..










share|improve this question























  • Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html

    – dglozano
    Nov 23 '18 at 18:35











  • Thank you my friend, but I don't have time to implement that :)

    – joe
    Nov 23 '18 at 18:38











  • So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now

    – dglozano
    Nov 23 '18 at 19:08











  • The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...

    – joe
    Nov 23 '18 at 19:11











  • Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end

    – dglozano
    Nov 23 '18 at 19:18














0












0








0








So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...



So this is the receiver:



static class ReceiveLogsThread implements Runnable {
private static final String TAG = "IoTReceiveLogsThread";
Message msgRead;

ReceiveLogsThread() {
}

public void run() {
byte rbuf = new byte[4096];
while (!Thread.currentThread().isInterrupted()) {
try {
int len = mSerial.readLog(rbuf, mSerialPortLog);
if (len > 0) {
// Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
// this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
receiveLogsHandler.sendMessage(this.msgRead);
}
} catch (NullPointerException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.currentThread().interrupt();
if (!mReadFlag) {
Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
}
}
}


As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation



   private static String printHex(byte bytes) {
Formatter formatter = new Formatter();

for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}


I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..



I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...



Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:



    private List<byte> logs = new ArrayList<>();

Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
public void handleMessage(Message msgRW) {
super.handleMessage(msgRW);
// logMessagesList.add(msgRW.obj.toString().toUpperCase());
// String message = msgRW.obj.toString().toUpperCase();

if(shouldCollectLogs) {
byte message = (byte) msgRW.obj;
logs.add(message);
}

....


So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..










share|improve this question














So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...



So this is the receiver:



static class ReceiveLogsThread implements Runnable {
private static final String TAG = "IoTReceiveLogsThread";
Message msgRead;

ReceiveLogsThread() {
}

public void run() {
byte rbuf = new byte[4096];
while (!Thread.currentThread().isInterrupted()) {
try {
int len = mSerial.readLog(rbuf, mSerialPortLog);
if (len > 0) {
// Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
// this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
receiveLogsHandler.sendMessage(this.msgRead);
}
} catch (NullPointerException e1) {
e1.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
Thread.currentThread().interrupt();
if (!mReadFlag) {
Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
}
}
}


As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation



   private static String printHex(byte bytes) {
Formatter formatter = new Formatter();

for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}


I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..



I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...



Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:



    private List<byte> logs = new ArrayList<>();

Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
public void handleMessage(Message msgRW) {
super.handleMessage(msgRW);
// logMessagesList.add(msgRW.obj.toString().toUpperCase());
// String message = msgRW.obj.toString().toUpperCase();

if(shouldCollectLogs) {
byte message = (byte) msgRW.obj;
logs.add(message);
}

....


So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..







java android arrays byte android-handler






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 23 '18 at 18:07









joejoe

1832824




1832824













  • Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html

    – dglozano
    Nov 23 '18 at 18:35











  • Thank you my friend, but I don't have time to implement that :)

    – joe
    Nov 23 '18 at 18:38











  • So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now

    – dglozano
    Nov 23 '18 at 19:08











  • The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...

    – joe
    Nov 23 '18 at 19:11











  • Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end

    – dglozano
    Nov 23 '18 at 19:18



















  • Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html

    – dglozano
    Nov 23 '18 at 18:35











  • Thank you my friend, but I don't have time to implement that :)

    – joe
    Nov 23 '18 at 18:38











  • So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now

    – dglozano
    Nov 23 '18 at 19:08











  • The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...

    – joe
    Nov 23 '18 at 19:11











  • Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end

    – dglozano
    Nov 23 '18 at 19:18

















Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html

– dglozano
Nov 23 '18 at 18:35





Even though I can't give you a detailed answer, I reccommend you to take a look to RxJava, especially to Flowables, Backpressure and Buffer. reactivex.io/intro.html

– dglozano
Nov 23 '18 at 18:35













Thank you my friend, but I don't have time to implement that :)

– joe
Nov 23 '18 at 18:38





Thank you my friend, but I don't have time to implement that :)

– joe
Nov 23 '18 at 18:38













So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now

– dglozano
Nov 23 '18 at 19:08





So, maybe I am not understanding correctly: What you want to do is to process the List<byte> in your Handler and transform it into a List<String> in which each string is the result of printToHex(bytes)? If that's the case, I'll post a simple answer now

– dglozano
Nov 23 '18 at 19:08













The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...

– joe
Nov 23 '18 at 19:11





The issue is: I'm receiving byte arrays and I immediately format them via pritHex method, which I think causes some delays and I lose some bytes which then causes messages to be unreadable... So I want to, istead of formating it to hex right away, add all those byte arrays into something (list of byte arrays) and then combine then in one in one big byte array (could be huge!). And then go through that big array and do formatting... In that time, receiver will be stopped and hopefully no bytes would be lost...

– joe
Nov 23 '18 at 19:11













Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end

– dglozano
Nov 23 '18 at 19:18





Yes, I understood the general idea, but I don't know if you need help with all of that or just the bytes formatting at the end

– dglozano
Nov 23 '18 at 19:18












2 Answers
2






active

oldest

votes


















0














Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.



private final static char hexArray = "0123456789ABCDEF".toCharArray(); 
public static String printToHex(byte bytes, int len) {
char hexChars = new char[len * 2];
for ( int j = 0; j < len; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}





share|improve this answer
























  • Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one

    – joe
    Nov 23 '18 at 20:20











  • @joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.

    – lionscribe
    Nov 23 '18 at 20:56













  • It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"

    – joe
    Nov 24 '18 at 2:20



















0














Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.



Alternative 1: Collect to List of String



List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.toList());// Collect the streams into one final list


Alternative 2: Collect to one String



String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
.map(bytes -> printHex(bytes)) // Map each byte array from byte to String
.collect(Collectors.joining()); // Concat the streams into one final String


You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.






share|improve this answer
























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53451247%2fhow-to-deal-with-huge-amount-of-byte-arrays%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.



    private final static char hexArray = "0123456789ABCDEF".toCharArray(); 
    public static String printToHex(byte bytes, int len) {
    char hexChars = new char[len * 2];
    for ( int j = 0; j < len; j++ ) {
    int v = bytes[j] & 0xFF;
    hexChars[j * 2] = hexArray[v >>> 4];
    hexChars[j * 2 + 1] = hexArray[v & 0x0F];
    }
    return new String(hexChars);
    }





    share|improve this answer
























    • Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one

      – joe
      Nov 23 '18 at 20:20











    • @joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.

      – lionscribe
      Nov 23 '18 at 20:56













    • It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"

      – joe
      Nov 24 '18 at 2:20
















    0














    Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.



    private final static char hexArray = "0123456789ABCDEF".toCharArray(); 
    public static String printToHex(byte bytes, int len) {
    char hexChars = new char[len * 2];
    for ( int j = 0; j < len; j++ ) {
    int v = bytes[j] & 0xFF;
    hexChars[j * 2] = hexArray[v >>> 4];
    hexChars[j * 2 + 1] = hexArray[v & 0x0F];
    }
    return new String(hexChars);
    }





    share|improve this answer
























    • Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one

      – joe
      Nov 23 '18 at 20:20











    • @joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.

      – lionscribe
      Nov 23 '18 at 20:56













    • It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"

      – joe
      Nov 24 '18 at 2:20














    0












    0








    0







    Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.



    private final static char hexArray = "0123456789ABCDEF".toCharArray(); 
    public static String printToHex(byte bytes, int len) {
    char hexChars = new char[len * 2];
    for ( int j = 0; j < len; j++ ) {
    int v = bytes[j] & 0xFF;
    hexChars[j * 2] = hexArray[v >>> 4];
    hexChars[j * 2 + 1] = hexArray[v & 0x0F];
    }
    return new String(hexChars);
    }





    share|improve this answer













    Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.



    private final static char hexArray = "0123456789ABCDEF".toCharArray(); 
    public static String printToHex(byte bytes, int len) {
    char hexChars = new char[len * 2];
    for ( int j = 0; j < len; j++ ) {
    int v = bytes[j] & 0xFF;
    hexChars[j * 2] = hexArray[v >>> 4];
    hexChars[j * 2 + 1] = hexArray[v & 0x0F];
    }
    return new String(hexChars);
    }






    share|improve this answer












    share|improve this answer



    share|improve this answer










    answered Nov 23 '18 at 19:47









    lionscribelionscribe

    2,1291714




    2,1291714













    • Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one

      – joe
      Nov 23 '18 at 20:20











    • @joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.

      – lionscribe
      Nov 23 '18 at 20:56













    • It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"

      – joe
      Nov 24 '18 at 2:20



















    • Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one

      – joe
      Nov 23 '18 at 20:20











    • @joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.

      – lionscribe
      Nov 23 '18 at 20:56













    • It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"

      – joe
      Nov 24 '18 at 2:20

















    Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one

    – joe
    Nov 23 '18 at 20:20





    Still feels like I'm losing some messages :/ I don't know if the buffer size is a problem, or bulk transfer timeout.. I know I should receive more of that one particular message, but I only received one

    – joe
    Nov 23 '18 at 20:20













    @joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.

    – lionscribe
    Nov 23 '18 at 20:56







    @joe Use the logCat to print how many bytes you get with call to printHex and without, and compare.

    – lionscribe
    Nov 23 '18 at 20:56















    It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"

    – joe
    Nov 24 '18 at 2:20





    It's tough to log it that way, because I don't know how many messages I should receive... I know I should look for a particular messages that should occur, actually more of the same message should be received after I send particular AT command...I either see one message or sometimes none.. But I should receive more... so that could only mean that some of them are "lost"

    – joe
    Nov 24 '18 at 2:20













    0














    Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.



    Alternative 1: Collect to List of String



    List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
    .map(bytes -> printHex(bytes)) // Map each byte array from byte to String
    .collect(Collectors.toList());// Collect the streams into one final list


    Alternative 2: Collect to one String



    String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
    .map(bytes -> printHex(bytes)) // Map each byte array from byte to String
    .collect(Collectors.joining()); // Concat the streams into one final String


    You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.






    share|improve this answer




























      0














      Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.



      Alternative 1: Collect to List of String



      List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
      .map(bytes -> printHex(bytes)) // Map each byte array from byte to String
      .collect(Collectors.toList());// Collect the streams into one final list


      Alternative 2: Collect to one String



      String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
      .map(bytes -> printHex(bytes)) // Map each byte array from byte to String
      .collect(Collectors.joining()); // Concat the streams into one final String


      You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.






      share|improve this answer


























        0












        0








        0







        Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.



        Alternative 1: Collect to List of String



        List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
        .map(bytes -> printHex(bytes)) // Map each byte array from byte to String
        .collect(Collectors.toList());// Collect the streams into one final list


        Alternative 2: Collect to one String



        String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
        .map(bytes -> printHex(bytes)) // Map each byte array from byte to String
        .collect(Collectors.joining()); // Concat the streams into one final String


        You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.






        share|improve this answer













        Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.



        Alternative 1: Collect to List of String



        List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
        .map(bytes -> printHex(bytes)) // Map each byte array from byte to String
        .collect(Collectors.toList());// Collect the streams into one final list


        Alternative 2: Collect to one String



        String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
        .map(bytes -> printHex(bytes)) // Map each byte array from byte to String
        .collect(Collectors.joining()); // Concat the streams into one final String


        You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 23 '18 at 19:31









        dglozanodglozano

        1,116221




        1,116221






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53451247%2fhow-to-deal-with-huge-amount-of-byte-arrays%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Paul Cézanne

            UIScrollView CustomStickyHeader Resize height generates problems when scroll is too fast

            Angular material date-picker (MatDatepicker) auto completes the date on focus out