一个Redis Cache实现（V2）

发布 by 猛禽 需要5分钟读完

更新说明

 

  • V2：缓存锁改用redis本身的setnx，原来用python的set不够安全。逻辑上也有少许修改——在第一次请求时如果没有缓存则等待，原来是不等待直接返回None，后续请求则与原来相同，直接返回旧值不等待。redis_cached装饰器的db参数变成函数，以便于动态创建的redis对象使用。

需求

 

应用中需要通过HTTP调用远程的数据，但是这个获取过程需要执行较长时间，而且这个数据本身的变化也不频繁，这种情况最适合用一个cache来优化。

前两年在做短链接实现的时候，曾经用最好的语言PHP做过一个Redis cache实现《一个简单的Redis应用（修订版）》，但那个毕竟是一个特定的实现，而且我现在需要的是python版。

这次的目标是需要实现一个比较通用的cache，支持各种数据类型，有超时更新机制，超时更新需要有锁（防止前文那个例子里发生过的问题）。

代码(py3)

 

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# python 3 required
from datetime import datetime
from functools import wraps
from traceback import format_exc
import hashlib
import pickle
import logging

from redis import StrictRedis as Redis


_author_ = 'raptor'


logger = logging.getLogger(_name_)


class RedisCache(object):
MAX_EXPIRES = 86400
LOCK_EXPIRES = 60
SERIALIZER = pickle


<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">__init__</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, name, host=<span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&#39;localhost&#39;</span>, port=<span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">6379</span>, db=<span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">0</span>, max_expires=MAX_EXPIRES)</span></span>:
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db = Redis(host=host, port=port, db=db)
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.name = name
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.max_expires = max_expires

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">_getkey</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, *keys)</span></span>:
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> <span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;%s:%s&quot;</span> % (<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.name, <span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;:&quot;</span>.join(keys))

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">_get_data</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, key)</span></span>:
    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.get(key)
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> None <span class="hljs-keyword" style="box-sizing: border-box;">if</span> result == b<span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&#39;None&#39;</span> <span class="hljs-keyword" style="box-sizing: border-box;">else</span> result

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">get</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, *keys)</span></span>:
    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>._get_data(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>._getkey(*keys))
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.SERIALIZER.loads(result) <span class="hljs-keyword" style="box-sizing: border-box;">if</span> result is <span class="hljs-keyword" style="box-sizing: border-box;">not</span> None <span class="hljs-keyword" style="box-sizing: border-box;">else</span> result

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">set</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, keys, value, ex=None)</span></span>:
    k = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>._getkey(*keys)
    v = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.SERIALIZER.dumps(value)
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.set(k, v, ex=ex)

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">delete</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, *keys)</span></span>:
    <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.delete(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>._getkey(*keys))

@staticmethod
<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">build_key</span><span class="hljs-params" style="box-sizing: border-box;">(name, *args, **kwargs)</span></span>:
    m = hashlib.md5()
    m.update(name.encode(<span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&#39;utf-8&#39;</span>))
    m.update(pickle.dumps(args))
    m.update(pickle.dumps(kwargs))
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> m.hexdigest()

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(153, 0, 0);">cached</span><span class="hljs-params" style="box-sizing: border-box;">(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>, key, func, ex=None)</span></span>:
    <span class="hljs-keyword" style="box-sizing: border-box;">if</span> ex is <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">None:</span>
        ex = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.max_expires
    min_ttl = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.max_expires - ex  <span class="hljs-comment" style="box-sizing: border-box; color: rgb(153, 153, 136); font-style: italic;"># ex &lt;= 0 : force refresh data</span>
    key = <span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;:&quot;</span>.join([<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.name, key])
    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>._get_data(key)
    lock_key = <span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;:&quot;</span>.join([<span class="hljs-string" style="box-sizing: border-box; color: rgb(221, 17, 68);">&quot;__lock__&quot;</span>, key])
    <span class="hljs-keyword" style="box-sizing: border-box;">if</span> <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.set(lock_key, <span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">1</span>, ex=<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.LOCK_EXPIRES, nx=True):
        <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">try:</span>
            ttl = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.ttl(key)
            <span class="hljs-keyword" style="box-sizing: border-box;">if</span> ttl is None <span class="hljs-keyword" style="box-sizing: border-box;">or</span> ttl &lt; <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">min_ttl:</span>
                result = func()
                <span class="hljs-keyword" style="box-sizing: border-box;">if</span> result is <span class="hljs-keyword" style="box-sizing: border-box;">not</span> <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">None:</span>
                    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.SERIALIZER.dumps(result)
                <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.set(key, result, ex=<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.max_expires)
        <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">finally:</span>
            <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.delete(lock_key)
    elif result is <span class="hljs-symbol" style="box-sizing: border-box; color: rgb(153, 0, 115);">None:</span>
        <span class="hljs-keyword" style="box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="box-sizing: border-box;">in</span> range(<span class="hljs-keyword" style="box-sizing: border-box;">self</span>.LOCK_EXPIRES * <span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">10</span>):
            time.sleep(<span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">0</span>.<span class="hljs-number" style="box-sizing: border-box; color: rgb(0, 128, 128);">1</span>)
            <span class="hljs-keyword" style="box-sizing: border-box;">if</span> <span class="hljs-keyword" style="box-sizing: border-box;">not</span> <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.db.exists(lock_key):
                <span class="hljs-keyword" style="box-sizing: border-box;">break</span>
        result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>._get_data(key)
    result = <span class="hljs-keyword" style="box-sizing: border-box;">self</span>.SERIALIZER.loads(result) <span class="hljs-keyword" style="box-sizing: border-box;">if</span> result is <span class="hljs-keyword" style="box-sizing: border-box;">not</span> None <span class="hljs-keyword" style="box-sizing: border-box;">else</span> None
    <span class="hljs-keyword" style="box-sizing: border-box;">return</span> result



def redis_cached(get_db, ex=None):
def decorator(fn):
@wraps(fn)
def wrapper(*args, **kwargs):
try:
key = RedisCache.build_key(fn._name_, *args, **kwargs)
return get_db().cached(key, lambda: fn(*args, **kwargs), ex)
except:
logger.error(format_exc())
return None
return wrapper
return decorator

用法

 

RedisCache本身也可以当一个Redis数据库对象使用，比如：

db = RedisCache('tablename', max_expires=3600)  #  tablename是一个是自定义的key前缀，可以用于当作表名使用。
# 最大超时时间(max_expires)仅供cached使用，使用set时，如果不指定超时时间则永不超时
db.set('aaa', {'key': 1234}, 7200)  # value可以是任何可序列化数据类型，比如字典，不指定超时则永不超时
db.get('aaa')['key']  # 结果为1234
db.delete('aaa')

但这个不是重点，重点是cached功能。对于慢速函数，加上db.cached以后，可以对函数调用的结果进行cache，在cache有效的情况下，大幅提高函数在反复调用时的性能。

下面是一个例子，具体见代码中的注释：

db = RedisCache('tablename')

def func(url, **kwargs):
result = requests.get("?".join([url, urlencode(kwargs)]))
return result


url = 'https://www.baidu.com/s'
t = time()
func(url, wd="测试")
print(time()-t)  # 较慢
t = time()
db.cached('test_cache', lambda: func(url, wd="测试"), 10)
print(time()-t)  # 第一次运行仍然较慢
t = time()
db.cached('test_cache', lambda: func(url, wd="测试"), 10)
print(time()-t)  # 从redis里读取cache很快
sleep(11)  # 等待到超时
t = time()
db.cached('test_cache', lambda: func(url, wd="测试"), 10)
print(time()-t)  # 超时后会再次执行func更新cache
t = time()
db.cached('test_cache_new', lambda: func(url, wd="新的测试"))
print(time()-t)  # 不同的调用参数用不同的key作cache
t = time()

因为对于不同的函数调用参数，函数可能有不同的返回结果，所以应该用不同的key进行cache。为简单起见，可以把函数签名做一个HASH，然后以此为KEY进行cache。最后把这个操作做成一个decorator，这样，只需要给函数加上这个decorator即可自动提供所需要的cache支持。

最终的简单用法如下：

db = RedisCache('tablename')

@redis_cached(lambda: db, 10)
def func(url, **kwargs):
result = requests.get("?".join([url, urlencode(kwargs)]))
return result


t = time()
func(url, wd="测试")
print(time()-t)
t = time()
func(url, wd="测试")
print(time()-t)
sleep(11)
t = time()
func(url, wd="测试")
print(time()-t)
t = time()
func(url, wd="新的测试")
print(time()-t)

是不是简单得多了。

推送到[go4pro.org]