$$ \newcommand{\problemdivider}{\begin{center}\large \bf\ldots\ldots\ldots\ldots\ldots\ldots\end{center}} \newcommand{\subproblemdivider}{\begin{center}\large \bf\ldots\ldots\end{center}} \newcommand{\pdiv}{\problemdivider} \newcommand{\spdiv}{\subproblemdivider} \newcommand{\ba}{\begin{align*}} \newcommand{\ea}{\end{align*}} \newcommand{\rt}{\right} \newcommand{\lt}{\left} \newcommand{\bp}{\begin{problem}} \newcommand{\ep}{\end{problem}} \newcommand{\bsp}{\begin{subproblem}} \newcommand{\esp}{\end{subproblem}} \newcommand{\bssp}{\begin{subsubproblem}} \newcommand{\essp}{\end{subsubproblem}} \newcommand{\atag}[1]{\addtocounter{equation}{1}\label{#1}\tag{\arabic{section}.\alph{subsection}.\alph{equation}}} \newcommand{\btag}[1]{\addtocounter{equation}{1}\label{#1}\tag{\arabic{section}.\alph{equation}}} \newcommand{\ctag}[1]{\addtocounter{equation}{1}\label{#1}\tag{\arabic{equation}}} \newcommand{\dtag}[1]{\addtocounter{equation}{1}\label{#1}\tag{\Alph{chapter}.\arabic{section}.\arabic{equation}}} \newcommand{\unts}[1]{\ \text{#1}} \newcommand{\textop}[1]{\operatorname{#1}} \newcommand{\textopl}[1]{\operatornamewithlimits{#1}} \newcommand{\prt}{\partial} \newcommand{\pderi}[3]{\frac{\prt^{#3}#1}{\prt #2^{#3}}} \newcommand{\deri}[3]{\frac{d^{#3}#1}{d #2^{#3}}} \newcommand{\del}{\vec\nabla} \newcommand{\exval}[1]{\langle #1\rangle} \newcommand{\bra}[1]{\langle #1|} \newcommand{\ket}[1]{|#1\rangle} \newcommand{\ham}{\mathcal{H}} \newcommand{\arr}{\mathfrak{r}} \newcommand{\conv}{\mathop{\scalebox{2}{\raisebox{-0.2ex}{$\ast$}}}} \newcommand{\bsm}{\lt(\begin{smallmatrix}} \newcommand{\esm}{\end{smallmatrix}\rt)} \newcommand{\bpm}{\begin{pmatrix}} \newcommand{\epm}{\end{pmatrix}} \newcommand{\bdet}{\lt|\begin{smallmatrix}} \newcommand{\edet}{\end{smallmatrix}\rt|} \newcommand{\bs}[1]{\boldsymbol{#1}} \newcommand{\uvec}[1]{\bs{\hat{#1}}} \newcommand{\qed}{\hfill$\Box$} $$
Tags:
  • python

    • Misc. notes

    • value_counts() omits NaN values by default

    Bad pandas

    groupby.apply() omits the group index if the indices of the rows being returned are identical to the indices of the input.

    Take the following 10 line dataframe:

    In [1]: import pandas as pd
In [2]: import numpy as np

In [3]: F = pd.DataFrame({ "a" : np.r_[0:10], "b" : np.r_[0:10]//5 })

   a  b
0  0  0
1  1  0
2  2  0
3  3  0
4  4  0
5  5  1
6  6  1
7  7  1
8  8  1
9  9  1

    If we group F by column b and take only the first two rows from each group, we get the group index b as the outer index, as expected:

    In [4]: F.groupby("b").apply(lambda x : x.iloc[:2])
Out[4]:
     a  b
b
0 0  0  0
  1  1  0
1 5  5  1
  6  6  1

    However, if we take all rows from each group, the outer group index is missing!

    In [5]: F.groupby("b").apply(lambda x : x.iloc[:])
Out[5]:
   a  b
0  0  0
1  1  0
2  2  0
3  3  0
4  4  0
5  5  1
6  6  1
7  7  1
8  8  1
9  9  1

In [6]: F.groupby("b").apply(lambda x : x)
Out[6]:
   a  b
0  0  0
1  1  0
2  2  0
3  3  0
4  4  0
5  5  1
6  6  1
7  7  1
8  8  1
9  9  1

    It seems that any function that preserves the order of the indices will cause the outer group index to disappear:

    In [7]: F.groupby("b").apply(lambda x : x + 1)
Out[7]:
    a  b
0   1  1
1   2  1
2   3  1
3   4  1
4   5  1
5   6  2
6   7  2
7   8  2
8   9  2
9  10  2

    Only functions that disrupt the order of the indices cause the outer group index to appear:

    In [8]: F.groupby("b").apply(lambda x : x.iloc[np.r_[0, 4, 1, 2, 3]])
Out[8]:
     a  b
b
0 0  0  0
  4  4  0
  1  1  0
  2  2  0
  3  3  0
1 5  5  1
  9  9  1
  6  6  1
  7  7  1
  8  8  1
  
In [9]: F.groupby("b").apply(lambda x : x.iloc[::-1])
Out[9]:
     a  b
b
0 4  4  0
  3  3  0
  2  2  0
  1  1  0
  0  0  0
1 9  9  1
  8  8  1
  7  7  1
  6  6  1
  5  5  1
    edit | history