重複標籤#
Index 物件不需要是唯一的；你可以有重複的行或欄位標籤。這一開始可能會有點令人困惑。如果你熟悉 SQL，你會知道行標籤類似於表格中的主鍵，而且你永遠不會想要在 SQL 表格中重複。但是，pandas 的其中一個角色是在資料進入下游系統之前，清理雜亂的真實世界資料。而真實世界資料有重複，即使是在應該唯一的欄位中。
本節說明重複標籤如何改變特定操作的行為，以及如何在操作期間防止重複發生，或是在重複發生時偵測到重複。
In [1]: import pandas as pd
In [2]: import numpy as np
重複標籤的後果#
一些 pandas 方法（例如 Series.reindex()）在有重複時無法運作。無法決定輸出，因此 pandas 會引發例外。
In [3]: s1 = pd.Series([0, 1, 2], index=["a", "b", "b"])
In [4]: s1.reindex(["a", "b", "c"])
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[4], line 1
----> 1 s1.reindex(["a", "b", "c"])
File ~/work/pandas/pandas/pandas/core/series.py:5144, in Series.reindex(self, index, axis, method, copy, level, fill_value, limit, tolerance)
5127 @doc(
5128 NDFrame.reindex, # type: ignore[has-type]
5129 klass=_shared_doc_kwargs["klass"],
(...)
5142 tolerance=None,
5143 ) -> Series:
-> 5144 return super().reindex(
5145 index=index,
5146 method=method,
5147 copy=copy,
5148 level=level,
5149 fill_value=fill_value,
5150 limit=limit,
5151 tolerance=tolerance,
5152 )
File ~/work/pandas/pandas/pandas/core/generic.py:5607, in NDFrame.reindex(self, labels, index, columns, axis, method, copy, level, fill_value, limit, tolerance)
5604 return self._reindex_multi(axes, copy, fill_value)
5606 # perform the reindex on the axes
-> 5607 return self._reindex_axes(
5608 axes, level, limit, tolerance, method, fill_value, copy
5609 ).__finalize__(self, method="reindex")
File ~/work/pandas/pandas/pandas/core/generic.py:5630, in NDFrame._reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy)
5627 continue
5629 ax = self._get_axis(a)
-> 5630 new_index, indexer = ax.reindex(
5631 labels, level=level, limit=limit, tolerance=tolerance, method=method
5632 )
5634 axis = self._get_axis_number(a)
5635 obj = obj._reindex_with_indexers(
5636 {axis: [new_index, indexer]},
5637 fill_value=fill_value,
5638 copy=copy,
5639 allow_dups=False,
5640 )
File ~/work/pandas/pandas/pandas/core/indexes/base.py:4429, in Index.reindex(self, target, method, level, limit, tolerance)
4426 raise ValueError("cannot handle a non-unique multi-index!")
4427 elif not self.is_unique:
4428 # GH#42568
-> 4429 raise ValueError("cannot reindex on an axis with duplicate labels")
4430 else:
4431 indexer, _ = self.get_indexer_non_unique(target)
ValueError: cannot reindex on an axis with duplicate labels
其他方法，例如索引，可能會產生非常令人驚訝的結果。通常使用純量進行索引會降低維度。使用純量切片 DataFrame 會傳回 Series。使用純量切片 Series 會傳回純量。但是，在有重複時，情況並非如此。
In [5]: df1 = pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "A", "B"])
In [6]: df1
Out[6]:
A A B
0 0 1 2
1 3 4 5
欄位中有重複。如果我們切片 'B'，我們會得到 Series
In [7]: df1["B"] # a series
Out[7]:
0 2
1 5
Name: B, dtype: int64
但是，切片 'A' 會傳回 DataFrame
In [8]: df1["A"] # a DataFrame
Out[8]:
A A
0 0 1
1 3 4
這也適用於行標籤
In [9]: df2 = pd.DataFrame({"A": [0, 1, 2]}, index=["a", "a", "b"])
In [10]: df2
Out[10]:
A
a 0
a 1
b 2
In [11]: df2.loc["b", "A"] # a scalar
Out[11]: 2
In [12]: df2.loc["a", "A"] # a Series
Out[12]:
a 0
a 1
Name: A, dtype: int64
重複標籤偵測#
你可以使用 Index（儲存列或欄位標籤）的 Index.is_unique 來檢查是否唯一
In [13]: df2
Out[13]:
A
a 0
a 1
b 2
In [14]: df2.index.is_unique
Out[14]: False
In [15]: df2.columns.is_unique
Out[15]: True
注意
對於大型資料集來說，檢查索引是否唯一會有點花時間。pandas 會快取這個結果，因此在同一個索引上重新檢查會非常快。
Index.duplicated() 會回傳一個布林陣列，指出標籤是否重複。
In [16]: df2.index.duplicated()
Out[16]: array([False, True, False])
可以用作布林篩選器來刪除重複列。
In [17]: df2.loc[~df2.index.duplicated(), :]
Out[17]:
A
a 0
b 2
如果你需要額外的邏輯來處理重複標籤，而不是只刪除重複項，在索引上使用 groupby() 是個常見的技巧。例如，我們將透過取得具有相同標籤的所有列的平均值來解決重複項。
In [18]: df2.groupby(level=0).mean()
Out[18]:
A
a 0.5
b 2.0
禁止重複標籤#
1.2.0 版的新功能。
如上所述，在讀取原始資料時，處理重複項是一個重要的功能。話雖如此，你可能希望避免在資料處理管線中引入重複項（來自 pandas.concat()、rename() 等方法）。Series 和 DataFrame 都透過呼叫 .set_flags(allows_duplicate_labels=False) 來禁止重複標籤。（預設是允許它們）。如果有重複標籤，將會引發例外狀況。
In [19]: pd.Series([0, 1, 2], index=["a", "b", "b"]).set_flags(allows_duplicate_labels=False)
---------------------------------------------------------------------------
DuplicateLabelError Traceback (most recent call last)
Cell In[19], line 1
----> 1 pd.Series([0, 1, 2], index=["a", "b", "b"]).set_flags(allows_duplicate_labels=False)
File ~/work/pandas/pandas/pandas/core/generic.py:507, in NDFrame.set_flags(self, copy, allows_duplicate_labels)
505 df = self.copy(deep=copy and not using_copy_on_write())
506 if allows_duplicate_labels is not None:
--> 507 df.flags["allows_duplicate_labels"] = allows_duplicate_labels
508 return df
File ~/work/pandas/pandas/pandas/core/flags.py:109, in Flags.__setitem__(self, key, value)
107 if key not in self._keys:
108 raise ValueError(f"Unknown flag {key}. Must be one of {self._keys}")
--> 109 setattr(self, key, value)
File ~/work/pandas/pandas/pandas/core/flags.py:96, in Flags.allows_duplicate_labels(self, value)
94 if not value:
95 for ax in obj.axes:
---> 96 ax._maybe_check_unique()
98 self._allows_duplicate_labels = value
File ~/work/pandas/pandas/pandas/core/indexes/base.py:715, in Index._maybe_check_unique(self)
712 duplicates = self._format_duplicate_message()
713 msg += f"\n{duplicates}"
--> 715 raise DuplicateLabelError(msg)
DuplicateLabelError: Index has duplicates.
positions
label
b [1, 2]
這適用於 DataFrame 的列標籤和欄標籤
In [20]: pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "B", "C"],).set_flags(
....: allows_duplicate_labels=False
....: )
....:
Out[20]:
A B C
0 0 1 2
1 3 4 5
此屬性可以使用 allows_duplicate_labels 檢查或設定，這表示物件是否可以有重複標籤。
In [21]: df = pd.DataFrame({"A": [0, 1, 2, 3]}, index=["x", "y", "X", "Y"]).set_flags(
....: allows_duplicate_labels=False
....: )
....:
In [22]: df
Out[22]:
A
x 0
y 1
X 2
Y 3
In [23]: df.flags.allows_duplicate_labels
Out[23]: False
DataFrame.set_flags() 可用於傳回新的 DataFrame，其屬性（例如 allows_duplicate_labels）設定為某個值
In [24]: df2 = df.set_flags(allows_duplicate_labels=True)
In [25]: df2.flags.allows_duplicate_labels
Out[25]: True
傳回的新 DataFrame 是與舊 DataFrame 相同資料的檢視。或屬性可以直接設定在同一個物件上
In [26]: df2.flags.allows_duplicate_labels = False
In [27]: df2.flags.allows_duplicate_labels
Out[27]: False
處理原始的雜亂資料時，您可能一開始會讀取雜亂的資料（可能具有重複標籤），刪除重複資料，然後禁止日後出現重複資料，以確保資料處理流程不會產生重複資料。
>>> raw = pd.read_csv("...")
>>> deduplicated = raw.groupby(level=0).first() # remove duplicates
>>> deduplicated.flags.allows_duplicate_labels = False # disallow going forward
在具有重複標籤的 Series 或 DataFrame 上設定 allows_duplicate_labels=False，或對禁止重複資料的 Series 或 DataFrame 執行會產生重複標籤的作業，將會引發 errors.DuplicateLabelError。
In [28]: df.rename(str.upper)
---------------------------------------------------------------------------
DuplicateLabelError Traceback (most recent call last)
Cell In[28], line 1
----> 1 df.rename(str.upper)
File ~/work/pandas/pandas/pandas/core/frame.py:5754, in DataFrame.rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
5623 def rename(
5624 self,
5625 mapper: Renamer | None = None,
(...)
5633 errors: IgnoreRaise = "ignore",
5634 ) -> DataFrame | None:
5635 """
5636 Rename columns or index labels.
5637
(...)
5752 4 3 6
5753 """
-> 5754 return super()._rename(
5755 mapper=mapper,
5756 index=index,
5757 columns=columns,
5758 axis=axis,
5759 copy=copy,
5760 inplace=inplace,
5761 level=level,
5762 errors=errors,
5763 )
File ~/work/pandas/pandas/pandas/core/generic.py:1139, in NDFrame._rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
1137 return None
1138 else:
-> 1139 return result.__finalize__(self, method="rename")
File ~/work/pandas/pandas/pandas/core/generic.py:6259, in NDFrame.__finalize__(self, other, method, **kwargs)
6252 if other.attrs:
6253 # We want attrs propagation to have minimal performance
6254 # impact if attrs are not used; i.e. attrs is an empty dict.
6255 # One could make the deepcopy unconditionally, but a deepcopy
6256 # of an empty dict is 50x more expensive than the empty check.
6257 self.attrs = deepcopy(other.attrs)
-> 6259 self.flags.allows_duplicate_labels = other.flags.allows_duplicate_labels
6260 # For subclasses using _metadata.
6261 for name in set(self._metadata) & set(other._metadata):
File ~/work/pandas/pandas/pandas/core/flags.py:96, in Flags.allows_duplicate_labels(self, value)
94 if not value:
95 for ax in obj.axes:
---> 96 ax._maybe_check_unique()
98 self._allows_duplicate_labels = value
File ~/work/pandas/pandas/pandas/core/indexes/base.py:715, in Index._maybe_check_unique(self)
712 duplicates = self._format_duplicate_message()
713 msg += f"\n{duplicates}"
--> 715 raise DuplicateLabelError(msg)
DuplicateLabelError: Index has duplicates.
positions
label
X [0, 2]
Y [1, 3]
此錯誤訊息包含重複的標籤，以及 Series 或 DataFrame 中所有重複資料（包括「原始資料」）的數字位置
重複標籤傳播#
一般來說，禁止重複資料是「持續性的」。在作業中會保留重複資料。
In [29]: s1 = pd.Series(0, index=["a", "b"]).set_flags(allows_duplicate_labels=False)
In [30]: s1
Out[30]:
a 0
b 0
dtype: int64
In [31]: s1.head().rename({"a": "b"})
---------------------------------------------------------------------------
DuplicateLabelError Traceback (most recent call last)
Cell In[31], line 1
----> 1 s1.head().rename({"a": "b"})
File ~/work/pandas/pandas/pandas/core/series.py:5081, in Series.rename(self, index, axis, copy, inplace, level, errors)
5074 axis = self._get_axis_number(axis)
5076 if callable(index) or is_dict_like(index):
5077 # error: Argument 1 to "_rename" of "NDFrame" has incompatible
5078 # type "Union[Union[Mapping[Any, Hashable], Callable[[Any],
5079 # Hashable]], Hashable, None]"; expected "Union[Mapping[Any,
5080 # Hashable], Callable[[Any], Hashable], None]"
-> 5081 return super()._rename(
5082 index, # type: ignore[arg-type]
5083 copy=copy,
5084 inplace=inplace,
5085 level=level,
5086 errors=errors,
5087 )
5088 else:
5089 return self._set_name(index, inplace=inplace, deep=copy)
File ~/work/pandas/pandas/pandas/core/generic.py:1139, in NDFrame._rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
1137 return None
1138 else:
-> 1139 return result.__finalize__(self, method="rename")
File ~/work/pandas/pandas/pandas/core/generic.py:6259, in NDFrame.__finalize__(self, other, method, **kwargs)
6252 if other.attrs:
6253 # We want attrs propagation to have minimal performance
6254 # impact if attrs are not used; i.e. attrs is an empty dict.
6255 # One could make the deepcopy unconditionally, but a deepcopy
6256 # of an empty dict is 50x more expensive than the empty check.
6257 self.attrs = deepcopy(other.attrs)
-> 6259 self.flags.allows_duplicate_labels = other.flags.allows_duplicate_labels
6260 # For subclasses using _metadata.
6261 for name in set(self._metadata) & set(other._metadata):
File ~/work/pandas/pandas/pandas/core/flags.py:96, in Flags.allows_duplicate_labels(self, value)
94 if not value:
95 for ax in obj.axes:
---> 96 ax._maybe_check_unique()
98 self._allows_duplicate_labels = value
File ~/work/pandas/pandas/pandas/core/indexes/base.py:715, in Index._maybe_check_unique(self)
712 duplicates = self._format_duplicate_message()
713 msg += f"\n{duplicates}"
--> 715 raise DuplicateLabelError(msg)
DuplicateLabelError: Index has duplicates.
positions
label
b [0, 1]
警告
這是一個實驗性功能。目前，許多方法無法傳遞 allows_duplicate_labels 值。在未來的版本中，預計每個使用或傳回一個或多個 DataFrame 或 Series 物件的方法都會傳遞 allows_duplicate_labels。