Pandas Dataframe selecting groups with minimal cardinality












2














I have a problem where I need to take groups of rows from a data frame where the number of items in a group exceeds a certain number (cutoff). For those groups, I need to take some head rows and the tail row.



I am using the code below



train = train[train.groupby('id').id.transform(len) > headRows]

groups = pd.concat([train.groupby('id').head(headRows),train.groupby('id').tail(1)]).sort_index()


This works. But the first line, it is very slow :(. 30 minutes or more.



Is there any way to make the first line faster ? If I do not use the first line, there are duplicate indices from the result of the second line, which messes up things.



Thanks in advance
Regards



Note:
My train data frame has around 70,000 groups of varying group size over around 700,000 rows . It actually follows from my other question as can be seen here Data processing with adding columns dynamically in Python Pandas Dataframe.
Jeff gave a great answer there, but it fails if the group size is less or equal to parameter I pass in head(parameter) when concatenating my rows as in Jeffs answer : In [31]: groups = concat.....






python pandas dataframe data-processing







share|improve this question





























    2














    I have a problem where I need to take groups of rows from a data frame where the number of items in a group exceeds a certain number (cutoff). For those groups, I need to take some head rows and the tail row.



    I am using the code below



    train = train[train.groupby('id').id.transform(len) > headRows]
    
groups = pd.concat([train.groupby('id').head(headRows),train.groupby('id').tail(1)]).sort_index()


    This works. But the first line, it is very slow :(. 30 minutes or more.



    Is there any way to make the first line faster ? If I do not use the first line, there are duplicate indices from the result of the second line, which messes up things.



    Thanks in advance
    Regards



    Note:
    My train data frame has around 70,000 groups of varying group size over around 700,000 rows . It actually follows from my other question as can be seen here Data processing with adding columns dynamically in Python Pandas Dataframe.
    Jeff gave a great answer there, but it fails if the group size is less or equal to parameter I pass in head(parameter) when concatenating my rows as in Jeffs answer : In [31]: groups = concat.....






    python pandas dataframe data-processing







    share|improve this question



























      2












      2








      2







      I have a problem where I need to take groups of rows from a data frame where the number of items in a group exceeds a certain number (cutoff). For those groups, I need to take some head rows and the tail row.



      I am using the code below



      train = train[train.groupby('id').id.transform(len) > headRows]
      
groups = pd.concat([train.groupby('id').head(headRows),train.groupby('id').tail(1)]).sort_index()


      This works. But the first line, it is very slow :(. 30 minutes or more.



      Is there any way to make the first line faster ? If I do not use the first line, there are duplicate indices from the result of the second line, which messes up things.



      Thanks in advance
      Regards



      Note:
      My train data frame has around 70,000 groups of varying group size over around 700,000 rows . It actually follows from my other question as can be seen here Data processing with adding columns dynamically in Python Pandas Dataframe.
      Jeff gave a great answer there, but it fails if the group size is less or equal to parameter I pass in head(parameter) when concatenating my rows as in Jeffs answer : In [31]: groups = concat.....






      python pandas dataframe data-processing







      share|improve this question















      I have a problem where I need to take groups of rows from a data frame where the number of items in a group exceeds a certain number (cutoff). For those groups, I need to take some head rows and the tail row.



      I am using the code below



      train = train[train.groupby('id').id.transform(len) > headRows]
      
groups = pd.concat([train.groupby('id').head(headRows),train.groupby('id').tail(1)]).sort_index()


      This works. But the first line, it is very slow :(. 30 minutes or more.



      Is there any way to make the first line faster ? If I do not use the first line, there are duplicate indices from the result of the second line, which messes up things.



      Thanks in advance
      Regards



      Note:
      My train data frame has around 70,000 groups of varying group size over around 700,000 rows . It actually follows from my other question as can be seen here Data processing with adding columns dynamically in Python Pandas Dataframe.
      Jeff gave a great answer there, but it fails if the group size is less or equal to parameter I pass in head(parameter) when concatenating my rows as in Jeffs answer : In [31]: groups = concat.....






      python pandas dataframe data-processing




      python pandas dataframe data-processing






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited May 23 '17 at 10:28









      Community♦

      11




      11










      asked May 18 '14 at 6:28









      Run2

      9721226




      9721226
























          1 Answer
          1






          active

          oldest

          votes


















          4














          Use groupby/filter:



          >>> df.groupby('id').filter(lambda x: len(x) > cutoff)


          This will just return the rows of your dataframe where the size of the group is greater than your cutoff. Also, it should perform quite a bit better. I timed filter here with a dataframe with 30,039 'id' groups and a little over 4 million observations:



          In [9]: %timeit df.groupby('id').filter(lambda x: len(x) > 12)
          
1 loops, best of 3: 12.6 s per loop





          share|improve this answer























          • Hey thanks Karl. Let me evaluate that. I will get back. Thanks
            – Run2
            May 18 '14 at 6:54






          • 1




            Hey Karl, worked. 4 mins now. Thanks.
            – Run2
            May 18 '14 at 7:04










          • @Karl D If u your take this question (and the original) and make a combined entry for the cookbook would be great (maybe add a little explanation of the problem). thanks
            – Jeff
            May 18 '14 at 10:57










          • Yeah, I can do that @Jeff.
            – Karl D.
            May 18 '14 at 18:35











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f23719203%2fpandas-dataframe-selecting-groups-with-minimal-cardinality%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          4














          Use groupby/filter:



          >>> df.groupby('id').filter(lambda x: len(x) > cutoff)


          This will just return the rows of your dataframe where the size of the group is greater than your cutoff. Also, it should perform quite a bit better. I timed filter here with a dataframe with 30,039 'id' groups and a little over 4 million observations:



          In [9]: %timeit df.groupby('id').filter(lambda x: len(x) > 12)
          
1 loops, best of 3: 12.6 s per loop





          share|improve this answer























          • Hey thanks Karl. Let me evaluate that. I will get back. Thanks
            – Run2
            May 18 '14 at 6:54






          • 1




            Hey Karl, worked. 4 mins now. Thanks.
            – Run2
            May 18 '14 at 7:04










          • @Karl D If u your take this question (and the original) and make a combined entry for the cookbook would be great (maybe add a little explanation of the problem). thanks
            – Jeff
            May 18 '14 at 10:57










          • Yeah, I can do that @Jeff.
            – Karl D.
            May 18 '14 at 18:35
















          4














          Use groupby/filter:



          >>> df.groupby('id').filter(lambda x: len(x) > cutoff)


          This will just return the rows of your dataframe where the size of the group is greater than your cutoff. Also, it should perform quite a bit better. I timed filter here with a dataframe with 30,039 'id' groups and a little over 4 million observations:



          In [9]: %timeit df.groupby('id').filter(lambda x: len(x) > 12)
          
1 loops, best of 3: 12.6 s per loop





          share|improve this answer























          • Hey thanks Karl. Let me evaluate that. I will get back. Thanks
            – Run2
            May 18 '14 at 6:54






          • 1




            Hey Karl, worked. 4 mins now. Thanks.
            – Run2
            May 18 '14 at 7:04










          • @Karl D If u your take this question (and the original) and make a combined entry for the cookbook would be great (maybe add a little explanation of the problem). thanks
            – Jeff
            May 18 '14 at 10:57










          • Yeah, I can do that @Jeff.
            – Karl D.
            May 18 '14 at 18:35














          4












          4








          4






          Use groupby/filter:



          >>> df.groupby('id').filter(lambda x: len(x) > cutoff)


          This will just return the rows of your dataframe where the size of the group is greater than your cutoff. Also, it should perform quite a bit better. I timed filter here with a dataframe with 30,039 'id' groups and a little over 4 million observations:



          In [9]: %timeit df.groupby('id').filter(lambda x: len(x) > 12)
          
1 loops, best of 3: 12.6 s per loop





          share|improve this answer














          Use groupby/filter:



          >>> df.groupby('id').filter(lambda x: len(x) > cutoff)


          This will just return the rows of your dataframe where the size of the group is greater than your cutoff. Also, it should perform quite a bit better. I timed filter here with a dataframe with 30,039 'id' groups and a little over 4 million observations:



          In [9]: %timeit df.groupby('id').filter(lambda x: len(x) > 12)
          
1 loops, best of 3: 12.6 s per loop






          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited May 18 '14 at 7:07

























          answered May 18 '14 at 6:51









          Karl D.

          7,0512827




          7,0512827












          • Hey thanks Karl. Let me evaluate that. I will get back. Thanks
            – Run2
            May 18 '14 at 6:54






          • 1




            Hey Karl, worked. 4 mins now. Thanks.
            – Run2
            May 18 '14 at 7:04










          • @Karl D If u your take this question (and the original) and make a combined entry for the cookbook would be great (maybe add a little explanation of the problem). thanks
            – Jeff
            May 18 '14 at 10:57










          • Yeah, I can do that @Jeff.
            – Karl D.
            May 18 '14 at 18:35


















          • Hey thanks Karl. Let me evaluate that. I will get back. Thanks
            – Run2
            May 18 '14 at 6:54






          • 1




            Hey Karl, worked. 4 mins now. Thanks.
            – Run2
            May 18 '14 at 7:04










          • @Karl D If u your take this question (and the original) and make a combined entry for the cookbook would be great (maybe add a little explanation of the problem). thanks
            – Jeff
            May 18 '14 at 10:57










          • Yeah, I can do that @Jeff.
            – Karl D.
            May 18 '14 at 18:35
















          Hey thanks Karl. Let me evaluate that. I will get back. Thanks
          – Run2
          May 18 '14 at 6:54




          Hey thanks Karl. Let me evaluate that. I will get back. Thanks
          – Run2
          May 18 '14 at 6:54




          1




          1




          Hey Karl, worked. 4 mins now. Thanks.
          – Run2
          May 18 '14 at 7:04




          Hey Karl, worked. 4 mins now. Thanks.
          – Run2
          May 18 '14 at 7:04












          @Karl D If u your take this question (and the original) and make a combined entry for the cookbook would be great (maybe add a little explanation of the problem). thanks
          – Jeff
          May 18 '14 at 10:57




          @Karl D If u your take this question (and the original) and make a combined entry for the cookbook would be great (maybe add a little explanation of the problem). thanks
          – Jeff
          May 18 '14 at 10:57












          Yeah, I can do that @Jeff.
          – Karl D.
          May 18 '14 at 18:35




          Yeah, I can do that @Jeff.
          – Karl D.
          May 18 '14 at 18:35


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f23719203%2fpandas-dataframe-selecting-groups-with-minimal-cardinality%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          "Incorrect syntax near the keyword 'ON'. (on update cascade, on delete cascade,)

          Image
          .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 2 total noob sorry any help greatly appreciated. I keep getting this "Incorrect syntax near the keyword 'ON'." heres my codes code one CREATE TABLE supplier ( supplierID INT PRIMARY KEY, ISBN INT FOREIGN KEY REFERENCES book(ISBN), price DECIMAL (5,2), ON UPDATE CASCADE, ON DELETE CASCADE, ); code two CREATE TABLE orders ( orderID INT PRIMARY KEY, customerID INT FOREIGN KEY REFERENCES customer(customerID), ISBN INT FOREIGN KEY REFERENCES book(ISBN), orderDate date, ON UPDATE CASCADE, ON DELETE CASCADE, ); Ive run out of stuff to try other than dumbing it down to uselessness. ...
          Read more

          Alcedinidae

          Image
          Alcedinidae Martin-pêcheur à dos bleu ( Alcedo azurea ) Classification (COI) Règne Animalia Embranchement Chordata Sous-embr. Vertebrata Classe Aves Ordre Coraciiformes Famille Alcedinidae Rafinesque, 1815 Les Alcédinidés ou Alcedinidae forment une famille d'oiseaux plus connus sous les noms de martins-pêcheurs et martins-chasseurs. Sommaire 1 Description 2 Répartition et habitat 3 Systématique 3.1 Liste alphabétique des genres 3.2 Liste des espèces 4 Chez les Anciens : l'alcyon 4.1 Étymologie (Littré) 4.2 Oiseau « fabuleux » ? 5 Notes 6 Articles connexes 7 Liens externes 7.1 Bibliographie Description | Ce sont des oiseaux compacts de taille petite à moyenne (10 à 46 cm), au bec droit et long, en forme de poignard. Leurs pattes sont courtes, et ils portent un plumage aux couleurs vives. Répartition et habitat | Cosmopolites, ils fréquentent surtout ...
          Read more

          Origin of the phrase “under your belt”?

          Image
          .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0; } 17 1 Today, I found myself discussing what students should have "under their belts" during a lecture, and I wondered to myself if there was some inappropriate undertone here that I might be unaware of. Question : What is the origin of the phrase "under your belt"? A google search revealed some webpages (e.g. [1]) that assert its origin is related to consumption of food -- once a meal has been eaten, it's under your belt (which is a relief). However, with such websites, I have no way to determine fact from "random guy on the internet making stuff up". etymology idioms ...
          Read more