熊猫：按时间间隔滚动平均值
- 2025-02-13 08:35:00
- admin 原创
- 53
问题描述：
我有一堆投票数据；我想计算 Pandas 滚动平均值，以根据三天窗口获得每天的估计值。根据这个问题，rolling_*函数根据指定数量的值而不是特定的日期时间范围来计算窗口。
我该如何实现这个功能？
示例输入数据：
polls_subset.tail(20)
Out[185]:
favorable unfavorable other
enddate
2012-10-25 0.48 0.49 0.03
2012-10-25 0.51 0.48 0.02
2012-10-27 0.51 0.47 0.02
2012-10-26 0.56 0.40 0.04
2012-10-28 0.48 0.49 0.04
2012-10-28 0.46 0.46 0.09
2012-10-28 0.48 0.49 0.03
2012-10-28 0.49 0.48 0.03
2012-10-30 0.53 0.45 0.02
2012-11-01 0.49 0.49 0.03
2012-11-01 0.47 0.47 0.05
2012-11-01 0.51 0.45 0.04
2012-11-03 0.49 0.45 0.06
2012-11-04 0.53 0.39 0.00
2012-11-04 0.47 0.44 0.08
2012-11-04 0.49 0.48 0.03
2012-11-04 0.52 0.46 0.01
2012-11-04 0.50 0.47 0.03
2012-11-05 0.51 0.46 0.02
2012-11-07 0.51 0.41 0.00
每个日期的输出只有一行。
解决方案 1：
同时，还添加了时间窗口功能。请参阅此链接。
In [1]: df = DataFrame({'B': range(5)})
In [2]: df.index = [Timestamp('20130101 09:00:00'),
...: Timestamp('20130101 09:00:02'),
...: Timestamp('20130101 09:00:03'),
...: Timestamp('20130101 09:00:05'),
...: Timestamp('20130101 09:00:06')]
In [3]: df
Out[3]:
B
2013-01-01 09:00:00 0
2013-01-01 09:00:02 1
2013-01-01 09:00:03 2
2013-01-01 09:00:05 3
2013-01-01 09:00:06 4
In [4]: df.rolling(2, min_periods=1).sum()
Out[4]:
B
2013-01-01 09:00:00 0.0
2013-01-01 09:00:02 1.0
2013-01-01 09:00:03 3.0
2013-01-01 09:00:05 5.0
2013-01-01 09:00:06 7.0
In [5]: df.rolling('2s', min_periods=1).sum()
Out[5]:
B
2013-01-01 09:00:00 0.0
2013-01-01 09:00:02 1.0
2013-01-01 09:00:03 3.0
2013-01-01 09:00:05 3.0
2013-01-01 09:00:06 7.0
解决方案 2：
那么像这样的事情怎么样：
首先将数据帧重新采样为 1D 间隔。这将取所有重复日期值的平均值。使用该fill_method选项填充缺失的日期值。接下来，将重新采样的帧传入pd.rolling_mean窗口为 3 且 min_periods=1 ：
pd.rolling_mean(df.resample("1D", fill_method="ffill"), window=3, min_periods=1)
favorable unfavorable other
enddate
2012-10-25 0.495000 0.485000 0.025000
2012-10-26 0.527500 0.442500 0.032500
2012-10-27 0.521667 0.451667 0.028333
2012-10-28 0.515833 0.450000 0.035833
2012-10-29 0.488333 0.476667 0.038333
2012-10-30 0.495000 0.470000 0.038333
2012-10-31 0.512500 0.460000 0.029167
2012-11-01 0.516667 0.456667 0.026667
2012-11-02 0.503333 0.463333 0.033333
2012-11-03 0.490000 0.463333 0.046667
2012-11-04 0.494000 0.456000 0.043333
2012-11-05 0.500667 0.452667 0.036667
2012-11-06 0.507333 0.456000 0.023333
2012-11-07 0.510000 0.443333 0.013333
更新：正如 Ben 在评论中指出的那样，pandas 0.18.0 的语法已经发生了变化。使用新的语法，它将是：
df.resample("1d").sum().fillna(0).rolling(window=3, min_periods=1).mean()
解决方案 3：
我刚刚遇到了同样的问题，但数据点间距不规则。重新采样在这里不是一个真正的选项。所以我创建了自己的函数。也许它对其他人也会有用：
from pandas import Series, DataFrame
import pandas as pd
from datetime import datetime, timedelta
import numpy as np
def rolling_mean(data, window, min_periods=1, center=False):
''' Function that computes a rolling mean
Parameters
----------
data : DataFrame or Series
If a DataFrame is passed, the rolling_mean is computed for all columns.
window : int or string
If int is passed, window is the number of observations used for calculating
the statistic, as defined by the function pd.rolling_mean()
If a string is passed, it must be a frequency string, e.g. '90S'. This is
internally converted into a DateOffset object, representing the window size.
min_periods : int
Minimum number of observations in window required to have a value.
Returns
-------
Series or DataFrame, if more than one column
'''
def f(x):
'''Function to apply that actually computes the rolling mean'''
if center == False:
dslice = col[x-pd.datetools.to_offset(window).delta+timedelta(0,0,1):x]
# adding a microsecond because when slicing with labels start and endpoint
# are inclusive
else:
dslice = col[x-pd.datetools.to_offset(window).delta/2+timedelta(0,0,1):
x+pd.datetools.to_offset(window).delta/2]
if dslice.size < min_periods:
return np.nan
else:
return dslice.mean()
data = DataFrame(data.copy())
dfout = DataFrame()
if isinstance(window, int):
dfout = pd.rolling_mean(data, window, min_periods=min_periods, center=center)
elif isinstance(window, basestring):
idx = Series(data.index.to_pydatetime(), index=data.index)
for colname, col in data.iterkv():
result = idx.apply(f)
result.name = colname
dfout = dfout.join(result, how='outer')
if dfout.columns.size == 1:
dfout = dfout.ix[:,0]
return dfout
# Example
idx = [datetime(2011, 2, 7, 0, 0),
datetime(2011, 2, 7, 0, 1),
datetime(2011, 2, 7, 0, 1, 30),
datetime(2011, 2, 7, 0, 2),
datetime(2011, 2, 7, 0, 4),
datetime(2011, 2, 7, 0, 5),
datetime(2011, 2, 7, 0, 5, 10),
datetime(2011, 2, 7, 0, 6),
datetime(2011, 2, 7, 0, 8),
datetime(2011, 2, 7, 0, 9)]
idx = pd.Index(idx)
vals = np.arange(len(idx)).astype(float)
s = Series(vals, index=idx)
rm = rolling_mean(s, window='2min')
解决方案 4：
user2689410 的代码正是我所需要的。提供我的版本（感谢 user2689410），由于一次性计算 DataFrame 中整行的平均值，因此速度更快。
希望我的后缀约定是可读的：_s：string、_i：int、_b：bool、_ser：Series 和 _df：DataFrame。如果发现多个后缀，类型可以是两者。
import pandas as pd
from datetime import datetime, timedelta
import numpy as np
def time_offset_rolling_mean_df_ser(data_df_ser, window_i_s, min_periods_i=1, center_b=False):
""" Function that computes a rolling mean
Credit goes to user2689410 at http://stackoverflow.com/questions/15771472/pandas-rolling-mean-by-time-interval
Parameters
----------
data_df_ser : DataFrame or Series
If a DataFrame is passed, the time_offset_rolling_mean_df_ser is computed for all columns.
window_i_s : int or string
If int is passed, window_i_s is the number of observations used for calculating
the statistic, as defined by the function pd.time_offset_rolling_mean_df_ser()
If a string is passed, it must be a frequency string, e.g. '90S'. This is
internally converted into a DateOffset object, representing the window_i_s size.
min_periods_i : int
Minimum number of observations in window_i_s required to have a value.
Returns
-------
Series or DataFrame, if more than one column
>>> idx = [
... datetime(2011, 2, 7, 0, 0),
... datetime(2011, 2, 7, 0, 1),
... datetime(2011, 2, 7, 0, 1, 30),
... datetime(2011, 2, 7, 0, 2),
... datetime(2011, 2, 7, 0, 4),
... datetime(2011, 2, 7, 0, 5),
... datetime(2011, 2, 7, 0, 5, 10),
... datetime(2011, 2, 7, 0, 6),
... datetime(2011, 2, 7, 0, 8),
... datetime(2011, 2, 7, 0, 9)]
>>> idx = pd.Index(idx)
>>> vals = np.arange(len(idx)).astype(float)
>>> ser = pd.Series(vals, index=idx)
>>> df = pd.DataFrame({'s1':ser, 's2':ser+1})
>>> time_offset_rolling_mean_df_ser(df, window_i_s='2min')
s1 s2
2011-02-07 00:00:00 0.0 1.0
2011-02-07 00:01:00 0.5 1.5
2011-02-07 00:01:30 1.0 2.0
2011-02-07 00:02:00 2.0 3.0
2011-02-07 00:04:00 4.0 5.0
2011-02-07 00:05:00 4.5 5.5
2011-02-07 00:05:10 5.0 6.0
2011-02-07 00:06:00 6.0 7.0
2011-02-07 00:08:00 8.0 9.0
2011-02-07 00:09:00 8.5 9.5
"""
def calculate_mean_at_ts(ts):
"""Function (closure) to apply that actually computes the rolling mean"""
if center_b == False:
dslice_df_ser = data_df_ser[
ts-pd.datetools.to_offset(window_i_s).delta+timedelta(0,0,1):
ts
]
# adding a microsecond because when slicing with labels start and endpoint
# are inclusive
else:
dslice_df_ser = data_df_ser[
ts-pd.datetools.to_offset(window_i_s).delta/2+timedelta(0,0,1):
ts+pd.datetools.to_offset(window_i_s).delta/2
]
if (isinstance(dslice_df_ser, pd.DataFrame) and dslice_df_ser.shape[0] < min_periods_i) or \n (isinstance(dslice_df_ser, pd.Series) and dslice_df_ser.size < min_periods_i):
return dslice_df_ser.mean()*np.nan # keeps number format and whether Series or DataFrame
else:
return dslice_df_ser.mean()
if isinstance(window_i_s, int):
mean_df_ser = pd.rolling_mean(data_df_ser, window=window_i_s, min_periods=min_periods_i, center=center_b)
elif isinstance(window_i_s, basestring):
idx_ser = pd.Series(data_df_ser.index.to_pydatetime(), index=data_df_ser.index)
mean_df_ser = idx_ser.apply(calculate_mean_at_ts)
return mean_df_ser
解决方案 5：
此示例似乎要求使用 @andyhayden 评论中所建议的加权平均值。例如，10 月 25 日有两次民意调查，10 月 26 日和 10 月 27 日各有一次民意调查。如果您只是重新采样然后取平均值，那么与 10 月 25 日的民意调查相比，这实际上会给 10 月 26 日和 10 月 27 日的民意调查赋予两倍的权重。
为了赋予每个民意调查相同的权重而不是赋予每一天相同的权重，您可以执行以下操作。
>>> wt = df.resample('D',limit=5).count()
favorable unfavorable other
enddate
2012-10-25 2 2 2
2012-10-26 1 1 1
2012-10-27 1 1 1
>>> df2 = df.resample('D').mean()
favorable unfavorable other
enddate
2012-10-25 0.495 0.485 0.025
2012-10-26 0.560 0.400 0.040
2012-10-27 0.510 0.470 0.020
这为您提供了进行基于民意调查的平均值（而不是基于天的平均值）的原始要素。与以前一样，民意调查是在 10 月 25 日进行平均的，但 10 月 25 日的权重也会被存储，并且是 10 月 26 日或 10 月 27 日权重的两倍，以反映 10 月 25 日进行了两次民意调查。
>>> df3 = df2 * wt
>>> df3 = df3.rolling(3,min_periods=1).sum()
>>> wt3 = wt.rolling(3,min_periods=1).sum()
>>> df3 = df3 / wt3
favorable unfavorable other
enddate
2012-10-25 0.495000 0.485000 0.025000
2012-10-26 0.516667 0.456667 0.030000
2012-10-27 0.515000 0.460000 0.027500
2012-10-28 0.496667 0.465000 0.041667
2012-10-29 0.484000 0.478000 0.042000
2012-10-30 0.488000 0.474000 0.042000
2012-10-31 0.530000 0.450000 0.020000
2012-11-01 0.500000 0.465000 0.035000
2012-11-02 0.490000 0.470000 0.040000
2012-11-03 0.490000 0.465000 0.045000
2012-11-04 0.500000 0.448333 0.035000
2012-11-05 0.501429 0.450000 0.032857
2012-11-06 0.503333 0.450000 0.028333
2012-11-07 0.510000 0.435000 0.010000
请注意，10 月 27 日的滚动平均值现在是 0.51500（民意调查加权），而不是 52.1667（日加权）。
另请注意，从 0.18.0 版本开始resample，API 已经发生了变化。rolling
滚动 (Pandas 0.18.0 中的新功能)
重新采样（pandas 0.18.0 中的新功能）
解决方案 6：
我发现当我尝试使用 window='1M' 时，user2689410 代码损坏了，因为业务月份的增量引发了这个错误：
AttributeError: 'MonthEnd' object has no attribute 'delta'
我添加了直接传递相对时间增量的选项，以便您可以对用户定义的时间段执行类似的事情。
谢谢您的指点，这是我的尝试 - 希望有用。
def rolling_mean(data, window, min_periods=1, center=False):
""" Function that computes a rolling mean
Reference:
http://stackoverflow.com/questions/15771472/pandas-rolling-mean-by-time-interval
Parameters
----------
data : DataFrame or Series
If a DataFrame is passed, the rolling_mean is computed for all columns.
window : int, string, Timedelta or Relativedelta
int - number of observations used for calculating the statistic,
as defined by the function pd.rolling_mean()
string - must be a frequency string, e.g. '90S'. This is
internally converted into a DateOffset object, and then
Timedelta representing the window size.
Timedelta / Relativedelta - Can directly pass a timedeltas.
min_periods : int
Minimum number of observations in window required to have a value.
center : bool
Point around which to 'center' the slicing.
Returns
-------
Series or DataFrame, if more than one column
"""
def f(x, time_increment):
"""Function to apply that actually computes the rolling mean
:param x:
:return:
"""
if not center:
# adding a microsecond because when slicing with labels start
# and endpoint are inclusive
start_date = x - time_increment + timedelta(0, 0, 1)
end_date = x
else:
start_date = x - time_increment/2 + timedelta(0, 0, 1)
end_date = x + time_increment/2
# Select the date index from the
dslice = col[start_date:end_date]
if dslice.size < min_periods:
return np.nan
else:
return dslice.mean()
data = DataFrame(data.copy())
dfout = DataFrame()
if isinstance(window, int):
dfout = pd.rolling_mean(data, window, min_periods=min_periods, center=center)
elif isinstance(window, basestring):
time_delta = pd.datetools.to_offset(window).delta
idx = Series(data.index.to_pydatetime(), index=data.index)
for colname, col in data.iteritems():
result = idx.apply(lambda x: f(x, time_delta))
result.name = colname
dfout = dfout.join(result, how='outer')
elif isinstance(window, (timedelta, relativedelta)):
time_delta = window
idx = Series(data.index.to_pydatetime(), index=data.index)
for colname, col in data.iteritems():
result = idx.apply(lambda x: f(x, time_delta))
result.name = colname
dfout = dfout.join(result, how='outer')
if dfout.columns.size == 1:
dfout = dfout.ix[:, 0]
return dfout
使用 3 天时间窗口来计算平均值的示例：
from pandas import Series, DataFrame
import pandas as pd
from datetime import datetime, timedelta
import numpy as np
from dateutil.relativedelta import relativedelta
idx = [datetime(2011, 2, 7, 0, 0),
datetime(2011, 2, 7, 0, 1),
datetime(2011, 2, 8, 0, 1, 30),
datetime(2011, 2, 9, 0, 2),
datetime(2011, 2, 10, 0, 4),
datetime(2011, 2, 11, 0, 5),
datetime(2011, 2, 12, 0, 5, 10),
datetime(2011, 2, 12, 0, 6),
datetime(2011, 2, 13, 0, 8),
datetime(2011, 2, 14, 0, 9)]
idx = pd.Index(idx)
vals = np.arange(len(idx)).astype(float)
s = Series(vals, index=idx)
# Now try by passing the 3 days as a relative time delta directly.
rm = rolling_mean(s, window=relativedelta(days=3))
>>> rm
Out[2]:
2011-02-07 00:00:00 0.0
2011-02-07 00:01:00 0.5
2011-02-08 00:01:30 1.0
2011-02-09 00:02:00 1.5
2011-02-10 00:04:00 3.0
2011-02-11 00:05:00 4.0
2011-02-12 00:05:10 5.0
2011-02-12 00:06:00 5.5
2011-02-13 00:08:00 6.5
2011-02-14 00:09:00 7.5
Name: 0, dtype: float64
解决方案 7：
为了保持基本性，我使用了一个循环和类似的东西来帮助你入门（我的索引是日期时间）：
import pandas as pd
import datetime as dt
#populate your dataframe: "df"
#...
df[df.index<(df.index[0]+dt.timedelta(hours=1))] #gives you a slice. you can then take .sum() .mean(), whatever
然后您可以在该切片上运行函数。您可以看到如何添加迭代器以使窗口的开头成为数据框索引中的第一个值以外的值，然后滚动窗口（例如，您也可以对开头使用 > 规则）。
请注意，对于超大数据或非常小的增量，这可能效率较低，因为切片可能会变得更加费力（对于我来说，对于数十万行数据和几列的数据，对于几周的每小时窗口来说，这已经足够好了）
解决方案 8：
检查你的索引是否真的datetime，而不是str
可能会有帮助：
data.index = pd.to_datetime(data['Index']).values
解决方案 9：
可视化滚动平均值以查看是否有意义。我不明白为什么在请求滚动平均值时使用总和。
df=pd.read_csv('poll.csv',parse_dates=['enddate'],dtype={'favorable':np.float,'unfavorable':np.float,'other':np.float})
df.set_index('enddate')
df=df.fillna(0)
fig, axs = plt.subplots(figsize=(5,10))
df.plot(x='enddate', ax=axs)
plt.show()
df.rolling(window=3,min_periods=3).mean().plot()
plt.show()
print("The larger the window coefficient the smoother the line will appear")
print('The min_periods is the minimum number of observations in the window required to have a value')
df.rolling(window=6,min_periods=3).mean().plot()
plt.show()
相关推荐
热门文章
项目管理软件有哪些？
热门标签
云禅道AD
