Playing with Couchbase: A Case Study

NoSQL in the context of Social Gaming Data

Created by Marc Trudel / @stelcheck

Intro

申し訳ございませんが。。。

About Wizcorp

Mobile Social Games
Internal middleware
Open-source projects

About me

Started as a Developer
Became System Administrator
Now Chief Technology Officer

About this story

It's about love
It will scare you
There is a lot to be learned

About this story: It's about love; we love Couchbase, and it has served us well. BUT! It will scare you. I HOPE it will scare you, because there are extremely important lessons to be learned. The lessons I will bring forward today are applicable not only to Couchbase, but to any NoSQL solution you might want to give a try. They are awesome, powerful and elegant solution to an large span of exciting challenges. But every weeks, we see online people reporting horror stories with them, and in most cases its their fault because they did not prepare their company to the shift, they didn't do their homeworks and they did not build and train their operations team to do a good work with what is, after all, a drastic change of paradigm.

Part 1

Couchbase for Social Games

The Early Days

PHP
MySQL as our prime datastore
Memcached
Lots of HTML5 goodness

But!

Real-time is hard
Asynchronous is hard
High Write/Read ratio
Scalability became an issue

Couchbase to the rescue!

Just like Memcached
Simple to operate
The future is bright

And most importantly

http://www.slideshare.net/renatko/couchbase-performance-benchmarking

It's fast

The Global Strategy

Couchbase + Node.js
node-memcached + Moxi
Store scalar values
Structured key patterns

The Strategy

Puppet
Graylog2
Nagios + Observium
Operate with admin panel

Part 2

Playtime's over, folks

It works!

Incredible performance
Very nice monitoring
Developers were happy

Then...

Key length/memory issues
Stat data does not persists
Weird errors with Moxi
node-memcached failure

And...

App-related data corruption
1 Node down, all game is down
Rebalance failures
Downtime and data recovery

Whaaaat??

What happened?

Rebalance bugs (1.8)
Crash/recovery strategies
Bad key/value storage
Persistent monitoring data

So some bugs asides....

We didn't do our homework properly

We assumed it would be simple

It was

We assumed it would be easy

It wasn't... Quite

Part 3

Strategies and Tools

Data storage

Reduce number of entries
Scalar values >> Documents
Better key generation strategy
Control over vbucket hash

Client library

Bug fixes to node-memcached
node-Couchbase contributions
Integrity check/repair scripts
node-archivist

Operations

Polyglot Persistence
Load test
Hot backups
Better rebalance strategy
NagiosForCouchbase