ycsb学习

发表于 | 分类于 | | 本文总阅读量 次
字数统计: 1.2k | 阅读时长 ≈ 6

主要总结Yahoo的数据库测试项目YCSB的使用（针对redis）。

github网址：https://github.com/brianfrankcooper/YCSB

安装

  • 需要安装

    • java
    • maven

  • 直接下载编译好的版本（不推荐）

    1
    2
    3
    curl -O --location https://github.com/brianfrankcooper/YCSB/releases/download/0.15.0/ycsb-0.15.0.tar.gz
    tar xfvz ycsb-0.15.0.tar.gz
    cd ycsb-0.15.0

  • 使用java或者maven编译

    1
    2
    3
    4
    5
    6
    git clone http://github.com/brianfrankcooper/YCSB.git
    cd YCSB
    # 只安装redis测试部分
    mvn -pl com.yahoo.ycsb:redis-binding -am clean package
    # 安装全部测试功能
    mvn clean package

概述

详细说明：https://github.com/brianfrankcooper/YCSB/wiki/Running-a-Workload

运行workload一共有六个部分

  • 设置数据库
  • 选择合适的DB interface
  • 选择合适的负载
  • 选择合适的runtime parameters
  • load选择好的workload
  • run选择好的workload

设置数据库

简要的说，就是创建名为usertable的表，因为ycsb默认的是对usertable进行相关操作。

而对于redis，则不需要相关的操作。

选择合适的DB interface

YCSB的操作是通过DB interface来实现的。最基本的DB interface是com.yahoo.ycsb.BasicDB，会将输出输出到System.out里。可以通过继承DB interface来自定义DB interface，也可以使用原有的DB interface。

选择合适的workload

当前版本提供了六种workload

Workload A: Update heavy workload

This workload has a mix of 50/50 reads and writes. An application example is a session store recording recent actions.

Workload B: Read mostly workload

This workload has a 95/5 reads/write mix. Application example: photo tagging; add a tag is an update, but most operations are to read tags.

Workload C: Read only

This workload is 100% read. Application example: user profile cache, where profiles are constructed elsewhere (e.g., Hadoop).

Workload D: Read latest workload

In this workload, new records are inserted, and the most recently inserted records are the most popular. Application example: user status updates; people want to read the latest.

Workload E: Short ranges

In this workload, short ranges of records are queried, instead of individual records. Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id).

Workload F: Read-modify-write

In this workload, the client will read a record, modify it, and write back the changes. Application example: user database, where user records are read and modified by the user or to record user activity.

自定义Workload

当然也可以自定义workload：https://github.com/brianfrankcooper/YCSB/wiki/Implementing-New-Workloads

一般常用的workload property如fieldcount，fieldlength，requestdistribution等。

全部properties如下：

The property files used with the core workload generator can specify values for the following properties:

  • fieldcount: the number of fields in a record (default: 10)
  • fieldlength: the size of each field (default: 100)
  • readallfields: should reads read all fields (true) or just one (false) (default: true)
  • readproportion: what proportion of operations should be reads (default: 0.95)
  • updateproportion: what proportion of operations should be updates (default: 0.05)
  • insertproportion: what proportion of operations should be inserts (default: 0)
  • scanproportion: what proportion of operations should be scans (default: 0)
  • readmodifywriteproportion: what proportion of operations should be read a record, modify it, write it back (default: 0)
  • requestdistribution: what distribution should be used to select the records to operate on – uniform, zipfian or latest (default: uniform)
  • maxscanlength: for scans, what is the maximum number of records to scan (default: 1000)
  • scanlengthdistribution: for scans, what distribution should be used to choose the number of records to scan, for each scan, between 1 and maxscanlength (default: uniform)
  • insertorder: should records be inserted in order by key (“ordered”), or in hashed order (“hashed”) (default: hashed)
  • operationcount: number of operations to perform. If set to zero then YCSB will run until maxexecutiontime is reached (which by default is indefinitely). Note that if your workload or command line parameters do not specify an operation count it will defualt to zero.
  • maxexecutiontime: maximum execution time in seconds. The benchmark runs until either the operation count has exhausted or the maximum specified time has elapsed, whichever is earlier. If unspecified the defualt is to run indefinitely.
  • table: the name of the table (default: usertable)
  • recordcount: number of records to load into the database initially (default: 0)
  • core_workload_insertion_retry_limit: number of attempts for any failed insert operation (default: 0)
  • core_workload_insertion_retry_interval: interval between retries, in seconds (default: 3)

选择合适的runtime parameter

主要是

  • -threads : the number of client threads.
  • -target : the target number of operations per second.
  • -s : status.十秒打印一次状态

load data

需要指定redis.host，redis.port。（可以指定redis.password和redis.cluster）

以上参数也可以在命令中指定，比如

1
./bin/ycsb load redis -s -P workloads/workloada -p "redis.host=127.0.0.1" -p "redis.port=6379"

run data

同理

1
./bin/ycsb run redis -s -P workloads/workloada -p "redis.host=127.0.0.1" -p "redis.port=6379"

一些注意事项

ycsb不支持对key的长度的修改

issue网址：https://github.com/brianfrankcooper/YCSB/issues/587

ycsb对key命名规则是两种，hashed模式会生成user加固定长度的一串hash值，而ordered模式会按照user加顺序的方式来命名。

1
2
insertorder=hashed      # user6284781860667377211, user8517097267634966620, user1820151046732198393
insertorder=ordered     # user1, user2, user3032

ycsb运行时错误：Read time out

错误类似于

1
2
3
4
5
6
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out

at redis.clients.jedis.Protocol.process(Protocol.java:79)
at redis.clients.jedis.Protocol.read(Protocol.java:131)
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:188)
at redis.clients.jedis.Jedis.sismember(Jedis.java:1266)

当数据量变大之后，延时可能会变成几秒，而Jedis默认的是2秒的超时限制。

修改redis的src文件夹的RedisClient.java

将

jedis = new Jedis(host, port);

修改为

jedis = new Jedis(host, port, 10000);

即从默认的2秒上升为10秒。

参考资料

  • ycsb github地址：https://github.com/brianfrankcooper/YCSB
  • ycsb 流程介绍：https://github.com/brianfrankcooper/YCSB/wiki/Running-a-Workload
  • ycsb workload介绍：https://github.com/brianfrankcooper/YCSB/wiki/Implementing-New-Workloads
  • ycsb 不支持对key长度的修改 issue网址：https://github.com/brianfrankcooper/YCSB/issues/587
  • ycsb 运行时错误：read time out 解决方法博客网址：https://github.com/brianfrankcooper/YCSB/issues/587