Key/value stores

This week, the challenge is to write the best key/value storage backend you can think of. It has to be persistent and the goal is for it to be capable of SET, GET, KEYS and DELETE operations, but it’s okay if you don’t have time to implement all of them. We prefer great code over lots of code.

You may implement it as a web service, CLI, Arduino serial adapter… You choose your environment. In the end, I think it would be funny to see lots of funky solutions.

To submit an entry, just put your work in a Gist and include a README to explain what it does and how it works (with code samples!). You have one week to enter, so good luck and have fun!

Prize

Once again, 6sync is offering $25 worth of credit for their hosting platform to this week’s winner!

This contest is finished

Congratulations to this week's winners! The entries and the contestant names are shown below.

Matt Sears

README.md

Gittr.rb

Git as a key-value store! Build with Grit, it supports SET, GET, KEYS, and DELETE operations. In addition, we can also get the change history of key/values.

And since it's Git, we can easily enhance it to include other awesome Git features such as branches, diffs, reverting, and more!

Example:

@store = Gittr.new(:repo => File.expand_path('..', __FILE__))
@store.clear  # Deletes all keys/values from the store

# SET
@store['lady'] = "gaga"

# GET
@store['lady'] #=> "gaga"

# KEYS
@store.keys  #=> ['lady']

# DELETE
@store.delete('lady') #=> 'gaga'

# LOG
@store.log('key')

# Produces:
[
 {"message"=>"all clear","committer"=>{"name"=>"Matt Sears", "email"=>"matt@mattsears.com"}, "committed_date"=>"..."},
 {"message"=>"set 'lady' ", "committer"=>{"name"=>"Matt Sears", "email"=>"matt@mattsears.com"}, "committed_date"=>"..."}
 {"message"=>"delete 'lady' ", "committer"=>{"name"=>"Matt Sears", "email"=>"matt@mattsears.com"}, "committed_date"=>"..."}
]

gittr.rb

require 'yaml'
require 'grit'

class Gittr

  def initialize(options = {})
    @options = options
    unless ::File.exists?(File.join(path,'.git'))
      Grit::Repo.init(path)
    end
  end

  # Add the value to the to the store
  #
  # Example
  #   @store.set('key', 'value')
  #
  # Returns nothing
  def set(key, value)
    save("set '#{key}'") do |index|
      index.add(key_for(key), encode(value))
    end
  end

  # Shortcut for #set
  #
  # Example:
  #  @store[key] = 'value'
  #
  def []=(key, value)
    set(key, value)
  end

  # Retrieve the value for the given key with a default value
  #
  # Example:
  #  @store.get(key)  #=> value
  #
  # Returns the object found in the repo matching the key
  def get(key, value = nil, *)
    if head && blob = head.commit.tree / key_for(key)
      decode(blob.data)
    end
  end

  # Shortcut for #get
  #
  # Example:
  #   @store['key']  #=> value
  #
  def [](key)
    get(key)
  end

  # Returns an array of key names contained in store
  #
  # Example:
  #  @store.keys  #=> ['key1', 'key2']
  #
  def keys
    head.commit.tree.contents.map{|blob| deserialize(blob.name) }
  end

  # Deletes commits matching the given key
  #
  # Example:
  #  @store.delete('key')
  #
  # Returns nothing
  def delete(key, *)
    self[key].tap do
      save("deleted #{key}") {|index| index.delete(key_for(key)) }
    end
  end

  # Deletes all contents of the store
  #
  # Returns nothing
  def clear
    save("all clear") do |index|
      if tree = index.current_tree
        tree.contents.each do |entry|
          index.delete(key_for(entry.name))
        end
      end
    end
  end

  # The commit log for the given key
  #
  # Example:
  #  @store.log('key') #=> [{"message"=>"Updated key"...}]
  #
  # Returns Array of commit data
  def log(key)
    git.log(branch, key_for(key)).map{ |commit| commit.to_hash }
  end

  # Find the key if exists in the git repo
  #
  # Example:
  #  @store.key? 'key'  #=> true
  #
  # Returns true if found; false if not found
  def key?(key)
    !(head && head.commit.tree / key_for(key)).nil?
  end

  private

  # Format the given key so that it ensures it's git worthy
  def key_for(key)
    key.is_a?(String) ? key : serialize(key)
  end

  # Given the file path, return a new Grit::Repo if found
  def git
    @git ||= Grit::Repo.new(path)
  end

  # The git branch to use for this store
  def branch
    @options[:branch] || 'master'
  end

  # Checks out the branch on the repo
  def head
    git.get_head(branch)
  end

  # Commits the the value into the git repository with the given commit message
  def save(message)
    index = git.index
    if head
      commit = head.commit
      index.current_tree = commit.tree
    end
    yield index
    index.commit(message, :parents => Array(commit), :head => branch) if index.tree.any?
  end

  # Converts the value to yaml format
  def encode(value)
    value.to_yaml
  end

  # Loads value as a Yaml structure
  def decode(value)
    YAML.load(value)
  end

  # Convert value to byte stream. This allows keys to be objects too
  def serialize(value)
    Marshal.dump(value)
  end

  # Converts value back to an object.
  def deserialize(value)
    Marshal.restore(value) rescue value
  end

  # Given that repo path set in the options, return the expanded file path
  def path(key = '')
    @path ||= File.join(File.expand_path(@options[:repo]), key)
  end

end

gittr_test.rb

require 'minitest/autorun'
require File.join(File.dirname(__FILE__), 'gittr.rb')

describe Gittr do

  @types = {
    "String" => ["lady", "gaga"],
    "Object" => [{:lady => :gaga}, {:gaga => :ohai}]
  }

  before do
    @store = Gittr.new(:repo => File.expand_path('..', __FILE__))
    @store.clear
  end

  @types.each do |type, (key, key2)|

    it "writes String values to keys" do
      @store[key] = "value"
      @store[key].must_equal "value"
    end

    it "reads from keys" do
      @store[key].must_be_nil
    end

    it "returns a list of keys" do
      @store[key] = "value"
      @store.keys.must_include(key)
    end

    it "guarantees that a different String value is retrieved" do
      value = "value"
      @store[key] = value
      @store[key].wont_be_same_as(value)
    end

    it "writes Object values to keys" do
      @store[key] = {:foo => :bar}
      @store[key].must_equal({:foo => :bar})
    end

    it "guarantees that a different Object value is retrieved" do
      value = {:foo => :bar}
      @store[key] = value
      @store[key].wont_be_same_as(:foo => :bar)
    end

    it "returns false from key? if a key is not available" do
      @store.key?(key).must_equal false
    end

    it "returns true from key? if a key is available" do
      @store[key] = "value"
      @store.key?(key).must_equal true
    end

    it "removes and return an element with a key from the store via delete if it exists" do
      @store[key] = "value"
      @store.delete(key).must_equal "value"
      @store.key?(key).must_equal false
    end

    it "returns nil from delete if an element for a key does not exist" do
      @store.delete(key).must_be_nil
    end

    it "removes all keys from the store with clear" do
      @store[key] = "value"
      @store[key2] = "value2"
      @store.clear
      @store.key?(key).wont_equal true
      @store.key?(key2).wont_equal true
    end

    it "does not run the block if the #{type} key is available" do
      @store[key] = "value"
      unaltered = "unaltered"
      @store.get(key) { unaltered = "altered" }
      unaltered.must_equal "unaltered"
    end

    it "stores #{key} values with #set" do
      @store.set(key, "value")
      @store[key].must_equal "value"
    end

    it "returns a list of commit history for the key" do
      @store.log(key).wont_be_empty
    end
  end

end

View full entry

Finished in 1st place with a final score of 3.9/5. (View the Gist)

Rico Sta. Cruz

z.md

Answers to questions you probably have:

The getting of keys are handled automatically because they're in /public.
Errors will automatically get a non-200 response.
Also, crap like http://localhost:4567/foo/../../ will already be trapped by Rack.
Running slow? Run it on a ramdisk (and make a cronjob for rsync for persistence) or buy an SSD.

0-readme.md

The simplest RESTy database ever

Oh yeah. It totally is.

Running it
- Type ruby persistence.rb — this runs on http://localhost:4567 by default.
Basic CRUD
- Adding keys: POST to /your/key/name/here
- Getting keys: GET /your/key/name/here
- Deleting keys: DELETE /your/key/name/here
Group operations
- Listing keys: GET /your/key/namespace/ (trailing slash)
- Deleting groups of keys: DELETE /your/key/namespace/ (trailing slash)

Try it out yourself!

require 'rest_client'
R = RestClient

puts "Adding /people/1          => " + R.post('http://localhost:4567/people/1', name: "Jason", age: "26")
puts "The contents of /people/1 => " + R.get('http://localhost:4567/people/1')
puts "Available keys:           => " + R.get('http://localhost:4567/people/')
puts "Deleting /people/1:       => " + R.delete('http://localhost:4567/people/1')
puts "Available keys:           => " + R.get('http://localhost:4567/people/')

#### Adding /people/1          => OK
#### The contents of /people/1 => {"name":"Jason","age":"26"}
#### Available keys:           => ["/people/1"]
#### Deleting /people/1:       => OK
#### Available keys:           => []

persistence.rb

require 'sinatra'
require 'json'
require 'fileutils'

before('*') { |path| @path = File.join(settings.public, path) }

post '*' do  # Create/update data
  params.delete "splat"
  FileUtils.mkdir_p File.dirname(@path)
  File.open(@path, 'w') { |f| f.write params.to_json } && "OK"
end

get '*/' do  # List keys
  Dir["#{@path}*"].select { |f| File.file? f }.map { |f| f.gsub settings.public, '' }.to_json
end

delete '*' do  # Delete key(s)
  FileUtils.rm_rf(@path) && "OK"
end

View full entry

Finished in 2nd place with a final score of 3.9/5. (View the Gist)

Sandro Turriate

Gemfile

source "http://rubygems.org"

# Specify your gem's dependencies in hash_proxy.gemspec
gemspec

Rakefile

require "bundler/gem_tasks"

hash_proxy.gemspec

# -*- encoding: utf-8 -*-
$:.push File.expand_path("../lib", __FILE__)
require "hash_proxy/version"

Gem::Specification.new do |s|
  s.name        = "hash_proxy"
  s.version     = HashProxy::VERSION
  s.authors     = ["Sandro Turriate"]
  s.email       = ["sandro.turriate@gmail.com"]
  s.homepage    = ""
  s.summary     = %q{TODO: Write a gem summary}
  s.description = %q{TODO: Write a gem description}

  s.rubyforge_project = "hash_proxy"

  s.files         = `git ls-files`.split("\n")
  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
  s.require_paths = ["lib"]

  # specify any dependencies here; for example:
  s.add_runtime_dependency "zmq"
  s.add_runtime_dependency "consistent_hashr"
  s.add_development_dependency "ruby-debug19"
end

.gitignore

*.gem
.bundle
Gemfile.lock
pkg/*
vendor

hash-proxy

#!/usr/bin/env ruby
require 'hash_proxy'

endpoint = ARGV[0] && !ARGV[0].empty? ? ARGV[0] : "tcp://127.0.0.1:6789"
HashProxy::Proxy.new(endpoint).serve

hash-proxy-node

#!/usr/bin/env ruby
require 'hash_proxy'

args = ARGV.dup
args.unshift('register') if args.empty?

endpoint = args[1] || "tcp://127.0.0.1:6789"
if args[0] == "register"
  HashProxy::Node.new.register(endpoint)
elsif args[0] == "serve"
  HashProxy::Node.new(endpoint).serve
else
  abort("Recognized commands are 'register' and 'serve'")
end

README.markdown

HashProxy

A distributed, persistent, key/value store providing language-agnostic storage-engines.

Requirements

Ruby 1.9.2 (Fibers)
ZeroMQ
ConsistentHashr gem

Example

Start a node to store data

bundle exec bin/hash-proxy-node
Start the proxy server

bundle exec bin/hash-proxy
Connect the client to the proxy

$ bundle console
>> c = HashProxy::Client.new
=> #<HashProxy::Client:0x00000100b7ff38 @endpoint="tcp://127.0.0.1:6789", @ctx=#<ZMQ::Context:0x00000100b7fee8>, @socket=#<ZMQ::Socket:0x00000100b7fec0>>
>> c.get('key')
=> nil
>> c.set('key', 'value')
=> "value"
>> c.get('key')
=> "value"
>> c.list
=> ["key"]
>> c.delete('key')
=> "value"
>> c.get('key')
=> nil
>> c.list
=> []

Start a new node

bundle exec bin/hash-proxy-node

Close the first node with CTRl-C, then check c.list; the array will be empty
Check the dump (persistence)

The 'dump' log is written out every second if 1000 entries have been made.

$ cat dump
The log gets restructured (truncated) every 60 seconds, leaving only the relevant changes.
Restarting the server when the log is present will redistribute the key-value pairs to all of the nodes present. It's always recommended to start the nodes before starting the proxy server. The server reditributes the keys almost immediately, and if a node is missing, the key will be stored on the next available node. When the original node returns, the server will search for the key there on the original node, and won't find the value.
Because zmq is the transport protocol, nodes can be written in any language (with a zmq driver). Check out node.py as an example (python node.py).
Don't need the persistence or distribution? Connect directly to a node. A small benchmark shows it to be twice as fast.

$ bundle exec bin/hash-proxy-node serve
$ bundle console
>> c = HashProxy::Client.new
>> c.set('foo', 'bar')
>> c.get('foo')

Thoughts

Having the nodes, and client all connect to a single process makes distribution very simple. Nodes can be added and removed at will without making changes to the client. The proxy server is the only IP/port that needs to be configured. While configuration is dead-simple, the proxy represents a single point of failure in our system. I've made no provisions for dealing with this problem, though it is solvable. I'm curious to know where the major speed bottle-neck is. Early on, I imagined the cache store (node) would be the bottle-neck, which is why I designed them to be language agnostic. I'd love to see a Ruby node compared to a C node. Though, when I connect directly to a node, skipping the proxy, it's clear that the proxy itself creates some latency, which is disappointing.

node.py

import zmq
import atexit
import socket

class Node:
  def __init__(self, endpoint=None):
    if endpoint:
      self.endpoint = endpoint
    else:
      self.endpoint = "tcp://127.0.0.1:%s" % self.next_available_port()
    self.ctx = zmq.Context()
    self.socket = self.ctx.socket(zmq.REP)
    self.store = {}
    def close_ctx():
      self.socket.close(); self.ctx.term()
    atexit.register(close_ctx)

  def serve(self):
    self.socket.bind(self.endpoint)
    print("Node starting on %s" % self.endpoint)
    while True:
      data = self.socket.recv()
      self.process(data)

  def process(self, data):
    properties = data.split(":", 3)
    instruction = properties[0]
    key = properties[1] if len(properties) > 1 else None
    value = properties[2] if len(properties) > 2 else None
    if instruction == "LIST":
      def translate(string):
        return string.replace(",", "%2C")
      keys = ",".join(map(translate, self.store.keys()))
      self.send("ACKLIST", keys)
    elif instruction == "SET":
      v = self.store[key] = value
      self.send("ACKSET", v)
    elif instruction == "GET":
      value = self.store[key] if self.store.has_key(key) else ""
      self.send("ACKGET", value)
    elif instruction == "DELETE":
      value = self.store.pop(key) if self.store.has_key(key) else ""
      self.send("ACKDELETE", value)

  def register(self, endpoint):
    client = self.ctx.socket(zmq.REQ)
    client.connect(endpoint)
    self.send("NODE", self.endpoint.replace(":", "%3A"), client)
    client.recv()
    def notify_close():
      self.send("NODEGONE", self.endpoint.replace(":", "%3A"), client)
      client.recv()
      client.close()
    atexit.register(notify_close)
    self.serve()

  def next_available_port(self):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind(("",0))
    s.listen(1)
    port = s.getsockname()[1]
    s.close()
    return port

  def send(self, key, value, socket=None):
    socket = socket if socket else self.socket
    return socket.send("%s:%s" % (key.upper(), value), zmq.NOBLOCK)

if __name__ == '__main__':
  Node().register('tcp://127.0.0.1:6789')

proxy.rb

module HashProxy
  class Proxy
    def initialize(endpoint)
      @endpoint = endpoint
      @ctx = ZMQ::Context.new
      @socket = @ctx.socket(ZMQ::REP)
      @nodes = {}
      @filename = 'dump'
      @file = File.open(@filename, 'a')
      @buffer = []
      @restructure_persistence_fork = Thread.new {}
      at_exit { @socket.close; @ctx.close; }
    end

    def read_fiber
      Fiber.new do
        while true
          if ZMQ.select([@socket], [], [], 0.5)
            data = @socket.recv
            process(data)
          end
          Fiber.yield
        end
      end
    end

    def persistence_fiber
      Fiber.new do
        while true
          if @buffer.size >= 1000 && not_restructuring?
            @file.write @buffer.join("\n")
            @file.puts
            @file.fsync
            @buffer.clear
          end
          Fiber.yield
        end
      end
    end

    def persistence_restructure_fiber
      Fiber.new do |tick|
        while true
          until tick > 60 && not_restructuring?
            tick += Fiber.yield
          end
          puts "Restructuring '#{@filename}' for greater efficiency."
          @file.fsync
          pid = fork { RestructurePersistence.new(@filename); exit! }
          @restructure_persistence_fork = Process.detach(pid)
          tick = 0
        end
      end
    end

    def tick_manager
      @tick_manager ||= TickManager.new
    end

    def not_restructuring?
      @restructure_persistence_fork.stop?
    end

    def recover_fiber
      Fiber.new do
        while @nodes.empty?
          Fiber.yield
        end
        puts "Attempting to recover from '#{@filename}'."
        File.open(@filename) do |f|
          f.each do |data|
            process(data.strip, true)
          end
        end
      end
    end

    def serve
      @socket.bind @endpoint
      puts "Server starting on #{@endpoint}"
      tick_manager.register(persistence_fiber)
      tick_manager.register(persistence_restructure_fiber)
      fibers = [read_fiber, recover_fiber, tick_manager]
      while true
        fibers.each do |fiber|
          if fiber.alive?
            fiber.resume
          else
            fibers.delete(fiber)
          end
        end
      end
    end

    def process(data, noreply=false)
      instruction, key, value = data.split(":", 3)
      case instruction
      when "NODE"
        key = URI.unescape(key)
        @nodes[key] = Client.new(key)
        ConsistentHashr.add_server(key, @nodes[key])
        send("ACK")
      when "NODEGONE"
        key = URI.unescape(key)
        @nodes.delete(key)
        ConsistentHashr.remove_server(key)
        send("ACK")
      when "LIST"
        aggregate_list
      when "SET"
        @buffer << data unless noreply
        client = ConsistentHashr.get(key)
        value = client.set(key, value)
        send("ACKSET", value) unless noreply
      when "GET"
        client = ConsistentHashr.get(key)
        send("ACKGET", client.get(key))
      when "DELETE"
        @buffer << data
        client = ConsistentHashr.get(key)
        send("ACKDELETE", client.delete(key))
      end
    end

    def aggregate_list
      lists = @nodes.values.map do |client|
        client.list_raw
      end.join(",")
      send("ACKLIST", lists)
    end

    private

    def send(key, value=nil, socket=@socket)
      socket.send("#{key.to_s.upcase}:#{value}", ZMQ::NOBLOCK)
    end

  end

  class TickManager
    def initialize
      @last_tick = Time.now
      @subscribers = []
    end

    def register(fiber)
      @subscribers << fiber
    end

    def fiber
      @fiber ||= Fiber.new do
        while true
          if tick > 1
            @subscribers.each {|f| f.resume(tick) if f.alive?}
            @last_tick = Time.now
          end
          Fiber.yield
        end
      end
    end

    def alive?
      fiber.alive?
    end

    def resume
      fiber.resume
    end

    def tick
      Time.now - @last_tick
    end
  end

end

version.rb

module HashProxy
  VERSION = "0.0.1"
end

node.rb

module HashProxy
  class Node
    def initialize(endpoint=nil)
      if endpoint
        @endpoint = endpoint
      else
        @endpoint = "tcp://127.0.0.1:#{next_available_port}"
      end
      @ctx = ZMQ::Context.new
      @socket = @ctx.socket(ZMQ::REP)
      at_exit { @socket.close; @ctx.close; }
      @store = {}
    end

    def serve
      @socket.bind @endpoint
      puts "Node starting on #{@endpoint}"
      while data = @socket.recv
        process(data)
      end
    end

    def process(data)
      instruction, key, value = data.split(":", 3)
      case instruction
      when "LIST"
        send("ACKLIST", @store.keys.map {|s| URI.escape(s.to_s, ',')}.join(","))
      when "SET"
        send("ACKSET", @store[key] = value)
      when "GET"
        send("ACKGET", @store[key])
      when "DELETE"
        send("ACKDELETE", @store.delete(key))
      end
    end

    def register(endpoint)
      client = @ctx.socket(ZMQ::REQ)
      client.connect(endpoint)
      send("NODE", URI.escape(@endpoint, ":"), client)
      client.recv
      at_exit do
        send("NODEGONE", URI.escape(@endpoint, ":"), client)
        client.recv
        client.close
      end
      serve
    end

    private

    def next_available_port
      server = TCPServer.new('127.0.0.1', 0)
      @port = server.addr[1]
    ensure
      server.close if server
    end

    def send(key, value, socket=@socket)
      socket.send("#{key.to_s.upcase}:#{value}", ZMQ::NOBLOCK)
    end

  end
end

restructure_persistence.rb

module HashProxy
  class RestructurePersistence
    def initialize(filename)
      @filename = filename
      @store = {}
      read
      write
    end

    def read
      File.open(@filename, 'r') do |f|
        f.each do |data|
          instruction, key, value = data.split(":", 3)
          case instruction
          when "SET"
            @store[key] = data
          when "DELETE"
            @store.delete(key)
          end
        end
      end
    end

    def write
      File.open(@filename, 'w') do |f|
        f.write @store.values.join
      end
    end
  end
end

hash_proxy.rb

require "hash_proxy/version"

module HashProxy
  require 'zmq'
  require 'consistent_hashr'
  require 'fiber'
  require 'socket'

  autoload 'Client', 'hash_proxy/client'
  autoload 'Node', 'hash_proxy/node'
  autoload 'Proxy', 'hash_proxy/proxy'
  autoload 'RestructurePersistence', 'hash_proxy/restructure_persistence'

  module ServerRemover
    def remove_server(_name)
      @number_of_replicas.times do |t|
        @circle.delete hash_key("#{_name}+#{t}")
      end
    end
  end
  ConsistentHashr.extend ServerRemover
end

client.rb

module HashProxy
  class Client
    def initialize(endpoint="tcp://127.0.0.1:6789")
      @endpoint = endpoint
      @ctx = ZMQ::Context.new
      @socket = @ctx.socket(ZMQ::REQ)
      at_exit { @socket.close; @ctx.close; }
      connect
    end

    def connect
      @socket.connect(@endpoint)
    end

    def list
      @socket.send("LIST:", ZMQ::NOBLOCK)
      l = process(@socket.recv)
      l = l.split(",").map{|s| URI.unescape(s)}
    end
    alias keys list

    def list_raw
      @socket.send("LIST:", ZMQ::NOBLOCK)
      process(@socket.recv)
    end

    def get(key)
      @socket.send("GET:#{key}", ZMQ::NOBLOCK)
      process(@socket.recv)
    end
    alias [] get

    def set(key, value)
      @socket.send("SET:#{key}:#{value}", ZMQ::NOBLOCK)
      process(@socket.recv)
    end
    alias []= set

    def delete(key)
      @socket.send("DELETE:#{key}", ZMQ::NOBLOCK)
      process(@socket.recv)
    end

    def process(data)
      instruction, value = data.split(":", 2)
      case instruction
      when "ACKLIST", "ACKSET"
        value
      when "ACKGET", "ACKDELETE"
        value unless value.empty?
      else
        raise "Unknown response: #{data}"
      end
    end
  end
end

.rvmrc

rvm use 1.9.2

View full entry

Finished in 3rd place with a final score of 3.8/5. (View the Gist)

Michael Tomer

Guardfile

guard 'shell' do
  watch(%r{^lib/(.+)\.rb$}) {|m| `ruby test/test_helper.rb` }
  watch(%r{^test/.+\.rb$})  {|m| `ruby test/test_helper.rb` }
end

README.markdown

Mnemosine

Mnemosine is a key/value store written in Ruby. It was created for the CodeBrawl competition located here: http://codebrawl.com/contests/key-value-stores

You can run the Mnemosine server by running "./mnemosine" from the root directory. There is also a Ruby adapter you can use. The API covers a small amount of what Redis can do, plus a tiny bit it can't.

Capabilities

Mnemosine's API is fairly similar to the Redis API, with a few differences. Like Redis, Mnemosine gives you several data types. You can use different functions depending on the data type.

Key

Key functions can be used on any value.

delete_all

Deletes every key and value pair in the entire database.

delete(k)

Deletes the key and its associated value.

keys

Lists all keys in the database.

exists(k)

Returns true if the key exists, false otherwise.

randomkey

Returns a random key from the database.

rename(oldkey, newkey)

Moves the value from one key to another, overwriting any data that be have been on the new key.

renamenx(oldkey, newkey)

Like rename, but returns an error if the new key already exists.

select_keys(src = nil)

Takes a block that is run on each key/value pair. Hopefully this can be expanded into a full map/reduce system at some point.

match_keys(regex, opts = "")

Takes a regular expression and returns any keys that match it.

Strings/Integers

These operations only work on strings or integers.

set(k, v)

Sets the key to the given value.

get(k)

Returns the value associated with the key.

append(k, v)

Only works with strings. Appends v to the end of the value associated with k.

incr(k)

Only works with integers. Adds one to the integer value associated with k.

decr(k)

Only works with integers. Subtracts one from the integer value associated with k.

incrby(k, v)

Only works with integers. Adds v to the integer value associated with k.

decrby(k, v)

Only works with integers. Subtracts v from the integer value associated with k.

Hashes

These functions only work with hashes.

hset(k, sk, v)

Assigns v to the sub-key sk.

hsetnx(k, sk, v)

Like hset, but it will not overwrite an existing key.

hget(k, sk)

Returns the value associated with the given key and sub-key.

hkeys(k)

Lists all keys in the hash.

hdel(k, sk)

Deletes the given sub-key from the hash, returning its value.

hexists(k, sk)

Checks to see if the given sub-key exists in the hash.

hlen(k)

Returns the number of keys in the hash.

hgetall(k)

Returns the entire hash.

hincr(k, sk)

Increments the integer stored in sk.

hdecr(k, sk)

Decrements the integer stored in sk.

hincrby(k, sk, v)

Adds v to the integer stored in sk.

hdecrby(k, sk, v)

Subtracts v from the integer stored in sk.

hmset(k, v)

Assigns v to the hash. Note that this overwrites everything. It would probably be preferable to merge it.

hmget(k)

Seems to be exactly the same as hgetall. I think I intended for this to take an array of sub-keys and just return those.

Arrays

This isn't very fleshed out yet. I intended to accomplish more with it, but my cat got very sick and the entire week has been spent dragging the poor guy to the emergency vet or various specialists. As such, there's not much here yet. I'd like to add some more functions, as well as length limited lists, which could be really cool for caching.

lset(k, sk, v)

Sets a value at the index specified by sk.

lget(k, sk)

Returns the value associated with sk.

lrange(k, start, stop)

Returns the values associated with the indexes between (and including) start and stop.

linsert(k, replace, v)

Inserts a value at replace, pushing any later keys further back. Redis is a little weird on this one, and I've emulated some of that weirdness. Redis wants you to specify whether you want to insert the value before or after the given key, which seems pointless. You're programmers, and you know how to add or subtract 1 from the index. However, Redis doesn't treat replace as an index, but as a value! That seems strange to me, and I'm not sure I like it. Still, that's how Redis does it, so I've made this function behave mostly the same.

Notes

Obviously Ruby is probably not a great choice for creating a database, as one typically wants very high performance. Instead, consider this to be a fun toy. It uses EventMachine on the server and passes JSON via persistent raw TCP connections. It also was also developed TDD style, so it could be good to look at if you're new to testing or Test::Unit.

I'll probably hack more features onto this as time permits. I'd like to get replication and sharding working, possibly a map/reduce system, and if I'm feeling really ambitious, perhaps some kind of indexed search.

There's currently no security on the database, and it only runs on localhost. As I said, this is just a fun little project, not a serious DB engine.

I'm putting this under the MIT license, so feel free to do whatever you want with it. My only request is that you don't use this as the basis for an entry into the upcoming CodeBrawl competition. While that would be hilarious, it would make me a sad panda.

Mnemosine offers a search system based on code execution. It's kind of like map reduce, but simpler and worse.

Requirements

Mnemosine is only compatible with Ruby 1.9. You'll need to install the following gems to use Mnemosine:

eventmachine
json
slop
sourcify

Run the following command to install sourcify:

gem install ruby_parser file-tail sourcify

mnemosine

#!/usr/bin/env ruby
require 'rubygems'
require 'slop'
require File.join(File.dirname(__FILE__), "lib", "server", "mnemosine")

opts = Slop.parse do
  on :p, :port, "Port the server will run on", true, default: 4291, :as => :integer
  on :s, :host, "Host the server will run on", true, default: 'localhost'
  on :h, :help, 'Show this message', :tail => true, do
    puts help
    exit
  end
end

Mnemosine::Server.new(opts)

test_helper.rb

require 'test/unit'
require 'FileUtils'

class MnemosineTest < Test::Unit::TestCase
  
  # When run in lib mode, the database just runs as a Ruby lib in your process rather
  # than starting EventMachine and taking TCP connections. Lib mode is only included
  # to make testing easier. It doesn't make any sense to use the DB in lib mode for
  # normal processing.
  #
  # In order to use Mnemosine properly, use the client. The client tests are in
  # the test_client.rb file.
  
  def setup
    @numeric_only_message = "That operation is only valid on: Fixnum"
    @string_only_message = "That operation is only valid on: String"
    @string_numeric_only_message = "That operation is only valid on: Fixnum, String"
    @hash_only_message = "That operation is only valid on: Hash"
    @list_only_message = "That operation is only valid on: Array"
  end
  
  def teardown
    @db.delete_all
  end
  
end

require File.expand_path(File.join File.dirname(__FILE__), 'test_key.rb')
require File.expand_path(File.join File.dirname(__FILE__), 'test_string.rb')
require File.expand_path(File.join File.dirname(__FILE__), 'test_hash.rb')
require File.expand_path(File.join File.dirname(__FILE__), 'test_list.rb')
require File.expand_path(File.join File.dirname(__FILE__), 'test_persistence.rb')

require File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "server", "mnemosine.rb"))

class ServerTest < MnemosineTest
  
  def setup
    @db = Mnemosine::Server.new(lib: true)
    super
  end
  
  include KeyTest
  include StringTest
  include HashTest
  include ListTest
  include PersistenceTest
end

require File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "client", "mnemosine.rb"))

class ClientTest < MnemosineTest
  
  def setup
    @db = Mnemosine::Client.new
    super
  end
  
  include KeyTest
  include StringTest
  include HashTest
  include ListTest
end

list.rb

class Mnemosine
  class Server
    
    def lset(k, sk, v)
      ensure_integer_key(sk) || ensure_array(@storage[k], empty: true) || (@storage[k] ||= []; @storage[k][sk] = v)
    end
    
    def lget(k, sk)
      ensure_integer_key(sk) || ensure_array(@storage[k]) || @storage[k][sk]
    end
    
    def lrange(k, start, stop)
      ensure_array(@storage[k]) || @storage[k][start..stop]
    end
    
    def linsert(k, replace, v)
      g = ensure_array(@storage[k])
      return g if g
      replace_idx = nil
      @storage[k].each.with_index do |val, idx|
        if val == replace
          replace_idx = idx
          break
        end
      end
      replace_idx ? @storage[k].insert(replace_idx, v) : {"error" => "Value to replace not found"}
    end
    
  end
end

test_hash.rb

module HashTest
  def test_hset
    @db.hset "foo", "bar", "baz"
    assert_equal "baz", @db.hget("foo", "bar")
  end
  
  def test_hset_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hset("foo", "bar", "baz"))
  end
  
  def test_hget_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hget("foo", "bar"))
  end
  
  def test_hkeys
    @db.hset "foo", "bar", "baz"
    @db.hset "foo", "lol", "cat"
    assert_equal ["bar", "lol"], @db.hkeys("foo").sort
  end
  
  def test_hkeys_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hkeys("foo"))
  end
  
  def test_hdel
    @db.hset "foo", "bar", "baz"
    @db.hset "foo", "lol", "cat"
    @db.hdel "foo", "bar"
    assert_equal ["lol"], @db.hkeys("foo")
  end
  
  def test_hdel_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hdel("foo", "bar"))
  end
  
  def test_hexists
    @db.hset "foo", "bar", "baz"
    assert_equal false, @db.hexists("foo", "lol")
    assert_equal true, @db.hexists("foo", "bar")
  end
  
  def test_hexists_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hexists("foo", "bar"))
  end
  
  def test_hlen
    @db.hset "foo", "bar", "baz"
    @db.hset "foo", "lol", "cat"
    assert_equal 2, @db.hlen("foo")
  end
  
  def test_hlen_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hlen("foo"))
  end
  
  def test_hgetall
    @db.hset "foo", "bar", "baz"
    @db.hset "foo", "lol", "cat"
    assert_equal({"bar" => "baz", "lol" => "cat"}, @db.hgetall("foo"))
  end
  
  def test_hgetall_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hgetall("foo"))
  end
  
  def test_hincr
    @db.hset "foo", "bar", 2
    @db.hincr "foo", "bar"
    assert_equal 3, @db.hget("foo", "bar")
  end
  
  def test_hincr_hash_string
    @db.hset "foo", "bar", "baz"
    assert_equal({"error" => @numeric_only_message}, @db.hincr("foo", "bar"))
  end
  
  def test_hincr_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hincr("foo", "bar"))
  end
  
  def test_hdecr
    @db.hset "foo", "bar", 2
    @db.hdecr "foo", "bar"
    assert_equal 1, @db.hget("foo", "bar")
  end
  
  def test_hdecr_hash_string
    @db.hset "foo", "bar", "baz"
    assert_equal({"error" => @numeric_only_message}, @db.hdecr("foo", "bar"))
  end
  
  def test_hdecr_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hdecr("foo", "bar"))
  end
  
  def test_hincrby
    @db.hset "foo", "bar", 2
    @db.hincrby "foo", "bar", 2
    assert_equal 4, @db.hget("foo", "bar")
  end
  
  def test_hincrby_hash_string
    @db.hset "foo", "bar", "baz"
    assert_equal({"error" => @numeric_only_message}, @db.hincrby("foo", "bar", 2))
  end
  
  def test_hincrby_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hincrby("foo", "baz", 2))
  end
  
  def test_hdecrby
    @db.hset "foo", "bar", 2
    @db.hdecrby "foo", "bar", 2
    assert_equal 0, @db.hget("foo", "bar")
  end
  
  def test_hdecrby_hash_string
    @db.hset "foo", "bar", "baz"
    assert_equal({"error" => @numeric_only_message}, @db.hdecrby("foo", "bar", 2))
  end
  
  def test_hdecrby_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hdecrby("foo", "bar", 2))
  end
  
  def test_hmset_and_get
    @db.hmset "foo", {"bar" => "baz", "lol" => "cat"}
    assert_equal({"bar" => "baz", "lol" => "cat"}, @db.hmget("foo"))
  end
  
  def test_hmset_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hmset("foo", {"bar" => "baz", "lol" => "cat"}))
  end
  
  def test_hmget_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hmget("foo"))
  end
  
  def test_hsetnx
    @db.hsetnx("foo", "bar", "baz")
    assert_equal("baz", @db.hget("foo", "bar"))
  end
  
  def test_hsetnx_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @hash_only_message}, @db.hsetnx("foo", "bar", "baz"))
  end
  
  def test_hsetnx_taken
    @db.hset "foo", "bar", "baz"
    assert_equal({"error" => "That key is already assigned"}, @db.hsetnx("foo", "bar", "lol"))
  end
  
end

test_string.rb

module StringTest
  def test_set
    @db.set "foo", "bar"
    assert_equal "bar", @db.get("foo")
  end
  
  def test_set_on_hash
    @db.hset "foo", "bar", "baz"
    assert_equal({"error" => @string_numeric_only_message}, @db.set("foo", "bar"))
  end
  
  def test_get_on_hash
    @db.hset "foo", "bar", "baz"
    assert_equal({"error" => @string_numeric_only_message}, @db.get("foo"))
  end
  
  def test_append_empty
    @db.append "foo", "bar"
    assert_equal "bar", @db.get("foo")
  end
  
  def test_append_present
    @db.set "foo", "lol"
    @db.append "foo", "cat"
    assert_equal "lolcat", @db.get("foo")
  end
  
  def test_append_number
    @db.set "foo", 42
    assert_equal({"error" => @string_only_message}, @db.append("foo", "bar"))
  end
  
  def test_incr
    @db.set "foo", 2
    @db.incr "foo"
    assert_equal 3, @db.get("foo")
  end
  
  def test_incr_string
    @db.set "foo", "bar"
    assert_equal({"error" => @numeric_only_message}, @db.incr("foo"))
  end
  
  def test_decr
    @db.set "foo", 2
    @db.decr "foo"
    assert_equal 1, @db.get("foo")
  end
  
  def test_decr_string
    @db.set "foo", "bar"
    assert_equal({"error" => @numeric_only_message}, @db.decr("foo"))
  end
  
  def test_incrby
    @db.set "foo", 2
    @db.incrby "foo", 2
    assert_equal 4, @db.get("foo")
  end
  
  def test_incrby_string
    @db.set "foo", "bar"
    assert_equal({"error" => @numeric_only_message}, @db.incrby("foo", 2))
  end
  
  def test_decrby
    @db.set "foo", 2
    @db.decrby "foo", 2
    assert_equal 0, @db.get("foo")
  end
  
  def test_decrby_string
    @db.set "foo", "bar"
    assert_equal({"error" => @numeric_only_message}, @db.decrby("foo", 2))
  end
  
end

test_persistence.rb

module PersistenceTest
  def test_save
    @db.set "foo", "bar"
    loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
    assert_equal true, @db.save(loc)
    assert File.exist?(File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))), "expected file to exist, but it does not"
    FileUtils.rm loc
  end

  def test_save_without_location
    @db.set "foo", "bar"
    assert_equal "No location given" , @db.save.values.first
  end

  def test_save_already_exists
    @db.set "foo", "bar"
    loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
    @db.save(loc)
    assert_equal "File already exists" , @db.save(loc).values.first
    FileUtils.rm loc
  end

  def test_restore
    @db.set "foo", "bar"
    loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
    @db.save(loc)
    @db.delete "foo"
    @db.restore loc
    assert_equal "bar", @db.get("foo")
    FileUtils.rm loc
  end

  def test_restore_no_file
    loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
    assert_equal "File does not exist", @db.restore(loc).values.first
  end

  def test_save_default_location
    loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
    @db.file_location = loc
    assert_equal true, @db.save
    FileUtils.rm loc
  end
end

test_key.rb

module KeyTest
  
  def test_delete_all
    @db.set "foo", "bar"
    @db.delete_all
    assert_nil @db.get("foo")
  end
  
  def test_delete
    @db.set "foo", "bar"
    assert_equal "bar", @db.delete("foo")
    assert_nil @db.get("foo")
  end
  
  def test_keys
    @db.set "foo", "bar"
    @db.set "lol", "cat"
    assert_equal ["foo", "lol"], @db.keys.sort
  end
  
  def test_exists
    @db.set "foo", "bar"
    assert_equal true, @db.exists("foo")
    assert_equal false, @db.exists("lol")
  end
  
  def test_randomkey
    @db.set "foo", "bar"
    @db.set "lol", "cat"
    assert ["foo", "lol"].include?(@db.randomkey), '"foo" or "lol" expected but was'
  end
  
  def test_rename
    @db.set "foo", "bar"
    @db.rename "foo", "baz"
    assert_nil @db.get "foo"
    assert_equal "bar", @db.get("baz")
  end
  
  def test_renamenx
    @db.set "foo", "bar"
    @db.set "baz", "wut"
    assert_equal({"error" => "That key is already assigned"}, @db.renamenx("foo", "baz"))
    assert_equal "bar", @db.get("foo")
    assert_equal "wut", @db.get("baz")
  end
  
  def test_select_keys
    @db.set "foo", 3
    @db.set "bar", 12
    @db.set "baz", 9
    @db.set "lol", 28
    @db.set "cat", 17
    assert_equal ["bar", "lol"], @db.select_keys {|k, v| v % 2 == 0}.sort
  end
  
  def test_match
    @db.set "foO", "bar"
    @db.set "bar", "wut"
    assert_equal ["foO"], @db.match_keys(/(\w)\1/i)
  end
  
end

persistence.rb

class Mnemosine
  class Server
    def save(location = nil)
      if location || @file_location
        persist(location || @file_location)
      else
        {"error" => "No location given"}
      end
    end
    
    def save!(location = nil)
      if location || @file_location
        persist!(location || @file_location)
      else
        {"error" => "No location given"}
      end
    end
    
    def restore(location = nil)
      if location || @file_location
        load(location || @file_location)
      else
        {"error" => "No location given"}
      end
    end
    
    def set_location(location)
      @file_location = location
    end
    
    def unset_location
      @file_location = nil
    end
    
    private
    
    def persist(loc)
      if File.exist?(loc)
        {error: "File already exists"}
      else
        persist!(loc)
      end
    end
    
    def persist!(loc)
      File.open(loc, "w") {|f| f.write @storage.to_json}
      true
    end
    
    def load(loc)
      if File.exist? loc
        @storage = JSON.parse(File.read(loc))
      else
        {"error" => "File does not exist"}
      end
    end
    
  end
end

process.rb

class Mnemosine
  class Server
    
    attr_accessor :file_location
    
    API_METHODS = %w[set get delete_all delete keys exists randomkey rename renamenx append incr decr incrby decrby
                     select_keys match_keys hset hget hmset hmget hlen hkeys hincr hincrby hdecr hdecrby hgetall
                     hexists hdel hsetnx lset lget linsert lrange]
    
    def initialize(args = {})
      @storage = new_storage
      @port = args[:p] || 4291
      @host = args[:s] || 'localhost'
      unless args[:lib]
        run_loop
      end
    end
    
    def new_storage
      {}
    end
    
    def receive_data(msg)
      return :send_nothing_back if msg == "\n"
      data = JSON.parse(msg)
      k = data.keys.first
      v = data.values.first
      if API_METHODS.include?(k)
        self.send(k, *v)
      else
        {"error" => "#{k} is not an API method"}
      end
    end
    
    def run_loop
      r = Runner
      r.setup(self)
      EventMachine.run do
        EventMachine.start_server @host, @port, r
        puts "Mnemosine listening at #{@host}:#{@port}"
      end
    end
    
    module Runner
      def self.setup(server)
        @@server = server
      end
      
      def receive_data(data)
        puts "Receiving data: #{data.inspect}"
        msg = @@server.receive_data(data)
        unless msg == :send_nothing_back
          puts "Sending msg: #{msg}"
          send_data({val: msg}.to_json + "\n")
          puts "Sent message"
          puts "\n"
        else
          puts "No response"
        end
      end
    end
    
  end
end

type_check.rb

class Mnemosine
  class Server
    
    def self.ensure_type(name, classes, opts = {})
      classes = [classes] unless classes.class == Array
      define_method "ensure_#{name}".to_sym, ->(value, args = {}) do
        if !args[:empty] && !value
          return {"error" => "Cannot perform that operation on an empty value"}
        elsif !value
          return nil
        elsif !classes.include?(value.class)
          return {"error" => (opts[:message] || "That operation is only valid on: #{classes.map {|c| c.to_s}.sort.join(", ")}")}
        end
      end
    end
    
    ensure_type :hash, Hash
    ensure_type :string, String
    ensure_type :numeric, Fixnum
    ensure_type :integer_key, Fixnum, :message => "Key must be an integer"
    ensure_type :string_or_numeric, [String, Fixnum]
    ensure_type :array, Array
    
  end
end

string.rb

class Mnemosine
  class Server
    
    def set(k, v)
      ensure_string_or_numeric(@storage[k], empty: true) || @storage[k] = v
    end
    
    def get(k)
      ensure_string_or_numeric(@storage[k], empty: true) || @storage[k]
    end
    
    def append(k, v)
      g = ensure_string(@storage[k], empty: true)
      return g if g
      if exists(k)
        @storage[k] += v
      else
        set k, v
      end
    end
    
    def incr(k)
      ensure_numeric(@storage[k]) || @storage[k] += 1
    end
    
    def decr(k)
      ensure_numeric(@storage[k]) || @storage[k] -= 1
    end
    
    def incrby(k, v)
      ensure_numeric(@storage[k]) || @storage[k] += v
    end
    
    def decrby(k, v)
      ensure_numeric(@storage[k]) || @storage[k] -= v
    end
    
  end
end

key.rb

class Mnemosine
  class Server
    
    def delete_all
      @storage = new_storage
    end
    
    def delete(k)
      @storage.delete k
    end
    
    def keys
      @storage.keys
    end
    
    def exists(k)
      !!@storage[k]
    end
    
    def randomkey
      @storage.keys[rand * @storage.keys.length]
    end
    
    def rename(old_key, new_key)
      @storage[new_key] = delete(old_key)
    end
    
    def renamenx(old_key, new_key)
      if exists(new_key)
        {"error" => "That key is already assigned"}
      else
        rename old_key, new_key
      end
    end
    
    def select_keys(src = nil)
      if src
        p = eval(src)
        @storage.select(&p).map(&:first)
      elsif block_given?
        @storage.select {|k, v| yield(k, v)}.map(&:first)
      end
    end
    
    def match_keys(regex, opts = "")
      if regex.class == String
        rgx = Regexp.new(regex, opts)
        @storage.keys.select {|k| k =~ rgx}
      else
        @storage.keys.select {|k| k =~ regex}
      end
    end
    
  end
end

hash.rb

class Mnemosine
  class Server
    
    def hset(k, sk, v)
      ensure_hash(@storage[k], empty: true) || (@storage[k] ||= {}; @storage[k][sk] = v)
    end
    
    def hget(k, sk)
      ensure_hash(@storage[k], empty: true) || (@storage[k] && @storage[k][sk])
    end
    
    def hkeys(k)
      ensure_hash(@storage[k], empty: true) || (@storage[k].keys)
    end
    
    def hdel(k, sk)
      ensure_hash(@storage[k], empty: true) || (@storage[k].delete sk)
    end
    
    def hexists(k, sk)
      ensure_hash(@storage[k]) || !!@storage[k][sk]
    end
    
    def hlen(k)
      ensure_hash(@storage[k]) || @storage[k].length
    end
    
    def hgetall(k)
      ensure_hash(@storage[k], empty: true) || @storage[k]
    end
    
    def hincr(k, sk)
      ensure_hash(@storage[k]) || ensure_numeric(@storage[k][sk]) || @storage[k][sk] += 1
    end
    
    def hdecr(k, sk)
      ensure_hash(@storage[k]) || ensure_numeric(@storage[k][sk]) || @storage[k][sk] -= 1
    end
    
    def hincrby(k, sk, v)
      ensure_hash(@storage[k]) || ensure_numeric(@storage[k][sk]) || @storage[k][sk] += v
    end
    
    def hdecrby(k, sk, v)
      ensure_hash(@storage[k]) || ensure_numeric(@storage[k][sk]) || @storage[k][sk] -= v
    end
    
    def hmset(k, v)
      ensure_hash(@storage[k], empty: true) || @storage[k] = v
    end
    
    def hmget(k)
      ensure_hash(@storage[k]) || @storage[k]
    end
    
    def hsetnx(k, sk, v)
      if !@storage[k]
        @storage[k] ||= {sk => v}
      else
        g = ensure_hash(@storage[k])
        if g
          g
        elsif !@storage[k][sk]
          @storage[k][sk] = v
        else
          {"error" => "That key is already assigned"}
        end
      end
    end
    
  end
end

messages.rb

require 'socket'
require 'json'

class Mnemosine
  class Client
    
    private
    
    def send_data(msg)
      @socket.puts msg.to_json
      JSON.parse(@socket.readline)["val"]
    end
    
    def self.api_methods(*mthds)
      mthds.each do |mthd|
        define_method mthd, ->(*args) do
          send_data({mthd => args})
        end
      end
    end
    public
    
    def initialize
      @socket = TCPSocket.open('localhost', 4291)
    end
    
    api_methods :set, :get, :delete_all, :delete, :keys, :exists, :rename, :renamenx, :incr,
                :decr, :incrby, :decrby, :append, :randomkey, :hset, :hget, :hmset, :hmget,
                :hlen, :hkeys, :hincr, :hincrby, :hdecr, :hdecrby, :hgetall, :hexists, :hdel,
                :hsetnx, :lset, :lget, :linsert, :lrange
    
    def select_keys(&blk)
      send_data({"select_keys" => blk.to_source})
    end
    
    def match_keys(regex)
      send_data({"match_keys" => [regex.source, regex.options]})
    end
    
  end
end

mnemosine.rb

require 'eventmachine'
require 'json'
require 'sourcify'

%w[type_check key string hash list persistence process].each do |file|
  require File.expand_path(File.join(File.dirname(__FILE__), "lib", "#{file}.rb"))
end

test_list.rb

module ListTest
  
  def test_lset
    @db.lset "foo", 0, "bar"
    assert_equal "bar", @db.lget("foo", 0)
  end
  
  def test_lset_string_key
    assert_equal({"error" => "Key must be an integer"}, @db.lset("foo", "bar", "baz"))
  end
  
  def test_lget_string_key
    assert_equal({"error" => "Key must be an integer"}, @db.lget("foo", "bar"))
  end
  
  def test_lset_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @list_only_message}, @db.lset("foo", 0, "bar"))
  end
  
  def test_lget_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @list_only_message}, @db.lget("foo", 0))
  end
  
  def test_lrange
    @db.lset "foo", 0, "bar"
    @db.lset "foo", 1, "baz"
    assert_equal ["bar", "baz"], @db.lrange("foo", 0, 1)
  end
  
  def test_lrange_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @list_only_message}, @db.lrange("foo", 0, 1))
  end
  
  def test_linsert
    @db.lset "foo", 0, "bar"
    @db.lset "foo", 1, "baz"
    @db.linsert "foo", "baz", "lol"
    assert_equal ["bar", "lol", "baz"], @db.lrange("foo", 0, 2)
  end
  
  def test_linsert_on_string
    @db.set "foo", "bar"
    assert_equal({"error" => @list_only_message}, @db.linsert("foo", "bar", "baz"))
  end
  
  def test_linsert_value_not_found
    @db.lset "foo", 0, "bar"
    assert_equal({"error" => "Value to replace not found"}, @db.linsert("foo", "lol", "cat"))
  end
  
end

View full entry

Finished in 4th place with a final score of 3.7/5. (View the Gist)

Scott Nelson

README.md

A minimalist append-only key/value store written in Ruby.

Ruby API

c = Collection.new('example.db')

c.set('hello', 'world')
c.get('hello') # => "world"

c.delete('hello')
c.get('hello') # => nil

c.set('foo', {'baz' => 'qux'})
c.get('foo')['baz'] # => "qux"

c.set('bar', [1,2,3])
c.get('bar').first # => 1

c.keys # => ["foo", "bar"]

Every update to the collection results in a JSON object being appended to the data file (example.db in the case above). Yes, even on deletes. Indices are kept in memory for speedy reads and the file can periodically be compacted to keep it from getting too unwieldy in size.

c.compact

Socket Interface

Included is a simple socket server interface built with EventMachine. Start listening on a given port and writing to the specified data file by running:

$ ruby server.rb 3456 example.db

Example client usage:

$ telnet localhost 3456
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get foo
{"baz":"qux"}
delete foo
OK
set hello world
OK
get hello
"world"
keys
["bar","hello"]
compact
OK
exit

server.rb

require 'json'

class Handlers
  class << self
    def handlers
      @handlers ||= {}
    end
    
    def on(command, respond = false, &blk)
      handlers[command.to_s] = Proc.new do |args|
        begin
          result = instance_exec(*args, &blk)
          reply(respond ? result.to_json : 'OK')
        rescue Exception
          reply('ERROR')
        end
      end
    end
    
    def handler(command)
      handlers[command] || Proc.new { reply('Unrecognized command') }
    end
  end
end

class CollectionHandlers < Handlers
  on(:get, true) { |key| @collection.get(key) }
  on(:set) { |key, *value| @collection.set(key, value.join(' ')) }
  on(:delete) { |key| @collection.delete(key) }
  on(:keys, true) { @collection.keys }
  on(:compact) { @collection.compact }
  on(:exit) { close_connection_after_writing }
end

module Server
  def initialize(collection)
    @collection = collection
  end
  
  def reply(data)
    send_data("#{data}\n")
  end
  
  def receive_data(data)
    command, *args = data.split(' ')
    handler = CollectionHandlers::handler(command.downcase)
    instance_exec(args, &handler)
  end
end

if __FILE__ == $0
  $: << File.dirname(__FILE__)
  
  require 'eventmachine'
  require 'store'
  
  EventMachine::run do
    port = ARGV[0]
    collection = Collection.new(ARGV[1])
    EventMachine::start_server('0.0.0.0', port, Server, collection)
    puts "Listening on port #{port}..."
  end
end

store.rb

require 'json'

class Storage
  attr_reader :path
  
  def initialize(path)
    @path = path
    File.open(path, 'a')
  end
  
  def read(position, length)
    JSON.parse(IO.read(@path, length, position))
  end
  
  def write(record)
    File.open(@path, 'a') do |f|
      [f.size, f.write("#{record.to_json}\n")]
    end
  end
  
  def each
    File.open(@path) do |f|
      position = 0
      f.each do |line|
        record = JSON.parse(line.chomp)
        length = line.length
        yield(record, position, length)
        position += length
      end
    end
  end
end

class Collection
  def initialize(path)
    @index = {}
    @storage = Storage.new(path)
    sync
  end
  
  def get(key)
    indexed = @index[key]
    if indexed
      @storage.read(indexed[:position], indexed[:length])['value']
    end
  end
  
  def set(key, value)
    position, length = @storage.write('key' => key, 'value' => value)
    index(key, position, length)
  end
  
  def delete(key)
    if @index[key]
      @storage.write('key' => key, 'deleted' => true)
      @index.delete(key)
    end
  end
  
  def keys
    @index.keys
  end
  
  def sync
    @index.clear
    @storage.each do |record, position, length|
      if record['deleted']
        @index.delete(record['key'])
      else
        index(record['key'], position, length)
      end
    end
  end
  
  def compact
    compacted = Storage.new(@storage.path + '.compact')
    indices = keys.map do |key|
      position, length = compacted.write('key' => key, 'value' => get(key))
      [key, position, length]
    end
    File.rename(compacted.path, @storage.path)
    @index.clear
    indices.each { |i| index(*i) }
  end
  
  private
  
  def index(key, position, length)
    @index[key] = { :position => position, :length => length }
  end
end

View full entry

Finished in 5th place with a final score of 3.6/5. (View the Gist)

Sebastian Staudt

README.md

Geist

Geist is a Git-backed key-value store written in Ruby. You may call it "Git Explicit Item Storage Tool" if you really want too.

Usage

require 'geist'

g = Geist.new '~/some/storage/path'
g.set :foo, 'bar'
g.set :baz, 123
g.set :name => 'geist', :platform => 'ruby'
g.delete :baz

g.keys                 #=> ["foo", "name", "platform"]
g.get :foo             #=> "bar"
g.get :baz             #=> nil
g.get :name, :platform #=> ["geist", "ruby"]

Internals

To be honest, the introduction was an outright fabrication. Geist is just a Ruby API to misuse Git as a simple key-value store. Git itself is a pretty good key-value store used to preserve blob (file), tree (directory), commit and tag objects.

The Ruby objects to store as values will be marshalled into Git blob objects. These objects are referenced with lightweight Git tags named by the given key.

Git will not double store duplicate values. Instead, the different key tags will refer the same Git object.

Caveats

As Geist uses Git tags as keys, only objects with a working #to_s method can be used as keys. Additionally, based on Git's ref naming rules, Geist rejects keys that can't be used as Git tag names, e.g. containing non-printable characters or backslashes.

License

This code is free software; you can redistribute it and/or modify it under the terms of the new BSD License. A copy of this license can be found in the LICENSE file.

Credits

Sebastian Staudt – koraktor(at)gmail.com

geist.rb

require 'fileutils'
require 'open4'

class Geist

  def initialize(path)
    @path = File.expand_path path

    if !File.exist? @path
      FileUtils.mkdir_p @path
      cmd 'init'
    elsif !cmd('ls-files').success?
      raise "#{@path} is not a Git repository."
    end
  end

  def delete(*keys)
    success = true

    keys.each do |key|
      if id_for(key).nil?
        success = false
      else
        cmd "tag -d '#{key}'"
      end
    end

    success
  end

  def get(*keys)
    return nil if keys.empty?

    values = []
    keys.each do |key|
      value = nil
      status = cmd "show '#{key}'" do |stdin, stdout|
        if select [stdout]
          blob = stdout.read.strip
          value = Marshal.load blob unless blob.empty?
        end
      end
      values << (status.success? ? value : nil)
    end

    keys.size == 1 ? values.first : values
  end

  def keys
    keys = nil
    cmd 'tag -l' do |stdin, stdout|
      keys = stdout.lines.to_a.map(&:strip) if select [stdout]
    end
    keys
  end

  def set(keys, value = nil)
    keys = { keys => value } unless keys.is_a? Hash

    keys.each do |key, val|
      if key.to_s.match /(?:[\W\s^~:?*\[\\]|\.\.|@\{|(?:\/|\.|\.lock)$|^$)/
        warn "Warning: Invalid key '#{key}'"
        return
      end

      delete key unless id_for(key).nil?

      id = nil
      cmd 'hash-object --stdin -w' do |stdin, stdout|
        stdin.write Marshal.dump(val)
        stdin.close
        id = stdout.read.strip if select [stdout]
      end

      cmd "tag -f '#{key}' #{id}"
    end
  end

  private

  def cmd(git_cmd, &block)
    cmd = "git --git-dir #{@path} #{git_cmd}"
    status = Open4::popen4 cmd do |pid, stdin, stdout, stderr|
      block.call stdin, stdout if block_given?

      stdin.close unless stdin.closed?
      stdout.close
      stderr.close
    end
    status
  end

  def id_for(key)
    id = nil
    status = cmd "rev-parse '#{key}'" do |stdin, stdout|
      id = stdout.read.strip if select [stdout]
    end
    status.success? ? id : nil
  end

end

View full entry

Finished in 6th place with a final score of 3.6/5. (View the Gist)

Vasiliy Ermolovich

README.md

Gingerbread

It's like very simple ActiveRecord for Sinatra but uses cookies instead of databases

Use cookies in Sinatra like a boss!

Example:

require './gingerbread'
require 'sinatra'

class Person < Gingerbread
  attribute :name
end

get '/create/:name' do
  person = Person.new(self)
  person.name = params[:name]
  person.save

  'Created!'
end

get '/find/:id' do
  person = Person.find(self, params[:id])

  "Hi, #{person.name}"
end

get '/delete/:id' do
  person = Person.find(self, params[:id])
  person.destroy(self)

  "#{person.name} has been deleted!"
end

gingerbread.rb

class Gingerbread

  def initialize(app, attributes = {})
    @@app = app
    @attributes = attributes
  end

  def save
    if @@app.request.cookies['gingerbread']
      data = Marshal.load(@@app.request.cookies['gingerbread'])
    else
      data = {}
    end

    data[class_name] ||= {}

    @attributes[:id] = next_id(data[class_name])

    data[class_name][@attributes[:id]] = self
    @@app.response.set_cookie 'gingerbread', :value => Marshal.dump(data), :path => '/'
  end

  def destroy(app)
    data = Marshal.load(app.request.cookies['gingerbread'])
    data[class_name].delete(@attributes[:id])
    app.response.set_cookie 'gingerbread', :value => Marshal.dump(data), :path => '/'
  end

  class << self
    def attribute(name, value = nil)
      define_method(name) do
        @attributes[name]
      end

      define_method("#{name}=") do |value|
        @attributes[name] = value
      end
    end

    def find(app, id)
      id = id.to_i
      if app.request.cookies['gingerbread']
        data = Marshal.load(app.request.cookies['gingerbread'])
        data[self.to_s.downcase.to_sym] ? data[self.to_s.downcase.to_sym][id] : nil
      else
        nil
      end
    end
  end

  private
    def class_name
      self.class.to_s.downcase.to_sym
    end

    def next_id(hash)
      id = hash.map{|a, b| a}.max
      (id || 0) + 1
    end
end

View full entry

Finished in 7th place with a final score of 3.4/5. (View the Gist)

rick

README.md

ZeroDB is a simple key/value store using ZeroMQ as the transport. It's pretty basic and synchronous right now. Keys are written to disk as-is.

Requirements

ruby 1.9.2

zmq gem - https://github.com/zeromq/rbzmq

brew install zeromq # 2.1.x
gem install zmq

Start a server

ruby server.rb

Start a Subscriber client to watch for changes

$ irb -r ./sub.rb
s = Subscriber.new
s.subscribe 'a'
s.on_set { |key| puts "#{key} :)" }
s.on_delete { |key| puts "#{key} :(" }
s.run

Start a key/value client

$ irb -r ./client.rb
c = Client.new
c.set 'abc', 123
c.set 'def', 456
c.get 'abc', 'def'
c.each { |key| puts key }
c.delete 'abc'
c.each { |key| puts key }
c.delete 'def'

server.rb

require 'zmq'
require 'fileutils'

class DB
  def initialize
    @root = File.join(File.dirname(__FILE__), 'data')
    FileUtils.mkdir_p @root
  end

  def get(key)
    name = file(key)
    File.exist?(name) ? IO.read(name) : nil
  end

  def set(key, value)
    File.open file(key), 'w' do |io|
      io << value
    end
  end

  def delete(key)
    FileUtils.rm_rf key
  end

  def each(&block)
    dir = Dir.new @root
    dir.each do |entry|
      next if entry =~ /^\./
      block.call entry
    end
  end

  def file(key)
    File.join @root, key
  end
end

class Server
  def initialize(db, context = nil)
    @db = db
    @context = context || ZMQ::Context.new
    @rep = @context.socket ZMQ::REP
    @rep.bind 'tcp://*:5555'
    @pub = @context.socket ZMQ::PUB
    @pub.bind 'tcp://*:5556'
  end
  
  def run
    while msg = @rep.recv
      cmd, *keys = msg.split ' '

      next unless valid_keys?(keys)

      method = "#{cmd.downcase}_command"
      if respond_to?(method)
        send(method, keys)
      else
        error "Invalid command: #{cmd}"
      end
    end
  rescue Interrupt
    puts "CLOSING"
    @rep.close
  end

  def get_command(keys)
    if keys.length > 0
      @rep.send 'OK', ZMQ::SNDMORE
      last = keys.pop
      keys.each do |k|
        @rep.send @db.get(k).to_s, ZMQ::SNDMORE
      end
      @rep.send @db.get(last).to_s
    else
      error "Need at least 1 key"
    end
  end

  def set_command(keys)
    if keys.length == 1
      @db.set keys[0], @rep.recv
      @rep.send 'OK'
      @pub.send "#{keys[0]} SET"
    else
      error "Need only 1 key"
    end
  end

  def delete_command(keys)
    keys.each do |k|
      @db.delete k
      @pub.send "#{k} DELETE"
    end

    @rep.send 'OK'
  end

  def keys_command(keys)
    @rep.send 'OK', ZMQ::SNDMORE
    @db.each do |key|
      @rep.send key, ZMQ::SNDMORE
    end
    @rep.send "DONE"
  end

  def error(msg)
    @rep.send "ERROR", ZMQ::SNDMORE
    @rep.send msg
  end

  def valid_keys?(keys)
    keys.each do |k|
      if k =~ /^[a-z0-9]+$/
        k.downcase!
      else
        error "Invalid key: #{k.inspect}"
        return false
      end
    end
    true 
  end
end

Server.new(DB.new).run

sub.rb

require 'zmq'

class Subscriber
  def initialize(context = nil)
    @context = context || ZMQ::Context.new
    @sub = @context.socket ZMQ::SUB
    @sub.connect 'tcp://127.0.0.1:5556'
    @on_set = @on_delete = nil
  end

  def subscribe(channel)
    @sub.setsockopt ZMQ::SUBSCRIBE, channel
  end

  def unsubscribe(channel)
    @sub.setsockopt ZMQ::UNSUBSCRIBE, channel
  end

  def on_set(&block)
    @on_set = block
  end

  def on_delete(&block)
    @on_delete = block
  end

  def run
    while msg = @sub.recv
      key, cmd = msg.split ' '
      block = case cmd
      when 'SET'    then @on_set
      when 'DELETE' then @on_delete
      end
      block ? block.call(key) : puts(key)
    end
  rescue Interrupt, IRB::Abort
    puts "CLOSING"
    @sub.close
  end
end

client.rb

require 'zmq'

class Client
  class ServerError < StandardError
  end

  attr_reader :context, :req
  def initialize(context = nil)
    @context = context || ZMQ::Context.new
    @req = @context.socket ZMQ::REQ
    @req.connect 'tcp://127.0.0.1:5555'
  end

  def get(*keys)
    @req.send "GET #{keys * " "}"
    error? || begin
      if keys.size == 1
        @req.recv
      else
        keys.map { |k| @req.recv }
      end
    end
  end

  def set(key, value)
    @req.send "SET #{key}", ZMQ::SNDMORE
    @req.send value.to_s
    error? || true
  end

  def delete(*keys)
    @req.send "DELETE #{keys * " "}"
    error? || true
  end

  def each(&block)
    @req.send "KEYS"
    error? || begin
      while key = @req.recv
        if key == "DONE"
          return
        else
          block.call key
        end
      end
    end
  end

  def error?
    if @req.recv != 'OK'
      raise ServerError, @req.recv
    end
  end

  def close
    @req.close
  end
end

View full entry

Finished in 8th place with a final score of 3.2/5. (View the Gist)

Federico Ravasio

README.md

A restful KV store

It relies on Sinatra to make everything pretty. Persistence is achieved with Marshal.

Methods

GET

curl "http://kvserver.dev/key1"
#=> value1

SET

curl -d "value3" "http://kvserver.dev/key3"
#=> value3

DELETE

curl -X delete "http://kvserver.dev/key3"
#=> value3

KEYS

curl "http://kvserver.dev"
#=> key1
    key2
    ...

test.rb

require './store'
require 'rack/test'

describe KVStore do
  include Rack::Test::Methods
  
  def app; KVStore; end
  
  let(:pairs){ {"test/key" => "test/value", "test/key2" => "test/value2"} }
  
  before(:each){ KVStore::Store.clear; KVStore::Store.merge! pairs }
  
  it 'should get a value' do
    pairs.each_pair do |k, v|
      get("/#{k}"){ |res| res.body.should == v }
    end
  end
  
  it 'should set a value' do
    post "/new/test", "5"
    last_response.body.should == "5"
  end
  
  it 'should remove a key' do
    pairs.each_pair do |k, v|
      delete("/#{k}"){ |res| res.body.should == v }
    end
  end
  
  it 'should get all the keys' do
    get("/"){ |res| res.body.should == pairs.keys.join("\n") }
  end
end

store.rb

require 'sinatra/base'

class KVStore < Sinatra::Base
  set :file, "store.msh"
  set :logging, true

  configure do
    Store = Marshal.load(File.read settings.file) rescue {}
  end
  
  helpers do
    def save_to_disk
      open(settings.file, "w") do |f|
        f.write Marshal.dump(Store)
      end
    end
  end

  before('/*') do
    @key = params[:splat].first
  end

  after '/*' do
    # Write to disk only on actual changes
    save_to_disk if env["REQUEST_METHOD"] =~ /post|delete/i and not body.empty?
  end

  get('/')     { body Store.keys.join("\n") }              # KEYS
  get('/*')    { body Store[@key] }                        # GET
  post('/*')   { body Store[@key] = params.flatten.first } # SET
  delete('/*') { body Store.delete(@key) }                 # DELETE
end

KVStore.run! if __FILE__ == $0

View full entry

Finished in 9th place with a final score of 3.2/5. (View the Gist)

Nelvin Driz

README.md

RuEDB (Ruby Evented Database)

Mainly based it out of redis. This basically supports basic key/value store that is accessible through through a REST API. It also stores it as yaml and has the option to encrypt the database given a key. (Although that needs manually changing the file atm.)

Environment:

ruby-1.9.2-p290
eventmachine
yajl
gibberish

Sample Usage:

# Run Server
$ ruby server.rb
Server started, Ruedb version 0.1.0
The server is now ready to accept connections on port 6187

# In another terminal
$ curl http://127.0.0.1:6187/values/name
$ curl http://127.0.0.1:6187/values/name -d "Some Value Here"
$ curl http://127.0.0.1:6187/values/name
$ curl http://127.0.0.1:6187/keys
$ curl -X DELETE http://127.0.0.1:6187/values/name

# I'm unsure though why curl returns "curl: (52) Empty reply from server" 
# when value is just one word. Value expected returns fine in the rest console I use
# in Google Chrome

database.rb

require 'singleton'
require 'gibberish'
require 'yajl'

module Ruedb
class Database
  include Singleton

class << self
  alias_method :connection, :instance
  def use(database = :default, key = nil)
    dump_database! if @database
    @database = get_path(database)
    @key      = key
    @@value_store, @@key_store = decrypt(key, @database)
  end

  def decrypt(key, database)
    return [Hash.new, Array.new] unless(File.exist?(database))
    file = File.open(database)
    raw_data = key ? Gibberish::AES.new(key).dec(file.read) : file.read
    file.close
    Yajl::Parser.new.parse(raw_data)
  end

  def dump_database!
    string = Yajl::Encoder.encode([@@value_store, @@key_store])
    encrypted_data = @key ? Gibberish::AES.new(@key).enc(string) : string
    file = File.open(@database, 'w+')
    file.write(encrypted_data)
    file.close
  end

  def get_path(database)
    return File.join('/', 'tmp', '.ruedb_store') if database.eql?(:default)
    # Check Ruedb::Config
    #return database
  end
end

  def initialize
    # Initialize Values
  end

  def key_store  ; @@key_store  ; end
  def value_store; @@value_store; end

  def set(key, value)
    key_store.push(key)
    length, library_key, unique_key = setup_values(key)

    setup_length(length)
    setup_library_key(length, library_key)
    value_store[length][library_key][unique_key] = value.to_s
  end

  def get(key)
    length, library_key, unique_key = setup_values(key)
    value_store[length][library_key][unique_key] || 'Key Not Found'
  rescue
    'Key Not Found'
  end

  def delete(key)
    key_store.delete(key)
    length, library_key, unique_key = setup_values(key)
    value = value_store[length][library_key].delete(unique_key)
    value ? {key => value} : 'Key Not Found'
  rescue
    'Key Not Found'
  end

  def keys
    key_store
  end

  def clear
    # TODO
  end

private

  def setup_values(key)
    key, length = key.to_s, key.length
    mid_length  = length / 2
    library_key = generate_library_key(mid_length, key)
    unique_key  = generate_unique_key(mid_length, length, key)
    [length.to_s, library_key.to_s, unique_key.to_s]
  end

  def generate_library_key(mid_length, key)
    key.slice(0, mid_length)
  end

  def generate_unique_key(mid_length, length, key)
    unique_key  = key.slice(mid_length, length);
  end

  def setup_length(length)
    value_store[length] ||= Hash.new
  end

  def setup_library_key(length, library_key)
    value_store[length][library_key] ||= Hash.new
  end

end
end

server.rb

$:.push File.expand_path("../", __FILE__)

require 'eventmachine'
require 'socket'

require 'database'
require 'version'

module Ruedb
module Server

  def post_init
    @port, @ip = Socket.unpack_sockaddr_in(get_peername)
    puts "Client from #{@ip}:#{@port} connected (currently serving #{EM.connection_count} clients)"
  end

  def receive_data(data)
    EM.defer(proc {
      match_data = data.match(/^(PUT|GET|POST|DELETE)\s([^\s]*).*\r\n\r\n(.*)/mx)
      raw_data, http_method, path, body = match_data.to_a
      route_path(http_method, path, body)
    }, proc { |result|
      puts "Result: #{result.inspect}"
      send_data(result)
      close_connection_after_writing
    })
  end

  def unbind
    puts "Client from #{@ip}:#{@port} disconnected"
  end

  def route_path(http_method, path, body)
    case path
    when /^\/values\/([^\/]*)\/?$/
      execute(http_method, $1, body)
    when /^\/keys\/?$/
      Ruedb::Database.connection.keys
    end
  end

  def execute(http_method, key, value = nil)
    case http_method
    when 'POST'
      puts "Store value #{value} into key #{key}"
      Ruedb::Database.connection.set(key, value)
    when 'DELETE'
      puts "Delete store with #{key}"
      Ruedb::Database.connection.delete(key)
    when 'GET'
      puts "Get value with key #{key}"
      Ruedb::Database.connection.get(key)
    end
  end

end
end

begin
  Ruedb::Database.use(:default)
  EventMachine.run do
    port = 6187
    EventMachine::add_periodic_timer(5) do
      Ruedb::Database.dump_database!
    end
    EventMachine.start_server '127.0.0.1', port, Ruedb::Server
    puts "Server started, Ruedb version #{Ruedb::VERSION}"
    puts "The server is now ready to accept connections on port #{port}"
  end
rescue SystemExit, Interrupt
  puts "# User requested shutdown..."
  puts "* Saving database snapshot before exiting."
  Ruedb::Database.dump_database!
  puts "* DB saved on disk"
  puts "RuEDB is now ready to exit, thanks!"
rescue Yajl::ParseError
  puts "Invalid Key to Access Database!"
end

View full entry

Finished in 10th place with a final score of 3.2/5. (View the Gist)

Joshua Hawxwell

README.md

Key-Value Store

Simplest key-value store that is usable (I think). Uses the stdlib PStore for storage which is not suited to storing large pieces of data, but isn't that the point of a key-value store?

GET / returns a json encoded array of keys (each as strings).

POST / creates a new key-value pair

GET /:key returns the value associated with the key

DELETE /:key deletes the key (and value)

Usage

To play around with it I'm using the htty gem (gem install htty).

$ ruby key_value.rb

$ htty http://0.0.0.0:4567
http://0.0.0.0:4567/> get
 200  OK -- 4 headers -- 2-character body
http://0.0.0.0:4567/> body
[]
http://0.0.0.0:4567/> query-set key value
http://0.0.0.0:4567/?key=value> post
 200  OK -- 3 headers -- 16-character body
http://0.0.0.0:4567/?key=value> query-clear
http://0.0.0.0:4567/> get
 200  OK -- 4 headers -- 7-character body
http://0.0.0.0:4567/> body
["key"]
http://0.0.0.0:4567/> cd /key
http://0.0.0.0:4567/key> get
 200  OK -- 4 headers -- 7-character body
http://0.0.0.0:4567/key> body
"value"
http://0.0.0.0:4567/key> delete
 200  OK -- 4 headers -- 7-character body
http://0.0.0.0:4567/key> get
 200  OK -- 4 headers -- 4-character body
http://0.0.0.0:4567/key> body
null
http://0.0.0.0:4567/key> cd /
http://0.0.0.0:4567/> get
 200  OK -- 4 headers -- 2-character body
http://0.0.0.0:4567/> body
[]
http://0.0.0.0:4567/>

key_value.rb

# http://www.ruby-doc.org/stdlib/libdoc/pstore/rdoc/classes/PStore.html
require 'pstore'
require 'sinatra'
require 'json'

PATH  = File.expand_path('~/pstore')
STORE = PStore.new(PATH)

get '/' do
  content_type :json
  STORE.transaction { STORE.roots.to_json }
end

post '/' do
  STORE.transaction do
    params.each {|k,v| STORE[k] = v }
  end
end

get '/:key' do
  content_type :json
  STORE.transaction { STORE[params[:key]].to_json }
end

delete '/:key' do
  STORE.transaction { STORE.delete(params[:key]).to_json }
end

key_value_test.rb

$: << File.dirname(__FILE__)

require 'key_value'
require 'minitest/autorun'
require 'rack/test'
require 'fileutils'

ENV['RACK_ENV'] = 'test'

class KeyValueTest < MiniTest::Unit::TestCase
  include Rack::Test::Methods
  
  def app
    Sinatra::Application
  end
  
  def teardown
    FileUtils.rm(PATH) if File.exist?(PATH)
  end
  
  def test_lists_keys
    get '/'
    assert last_response.ok?
    assert_equal '[]', last_response.body
    
    post '/?key=value'
    get '/'
    assert_equal '["key"]', last_response.body
  end
  
  def test_sets_and_gets_key_values
    post '/?key=value'
    assert last_response.ok?
    
    get '/key'
    assert last_response.ok?
    assert_equal '"value"', last_response.body
  end
  
  def test_deletes_key
    post '/?a=b'
    get '/a'
    assert_equal '"b"', last_response.body
    
    delete '/a'
    assert last_response.ok?
    get '/a'
    assert_equal 'null', last_response.body
  end

end

View full entry

Finished in 11th place with a final score of 3.0/5. (View the Gist)

Stefan
README.md
README

This is a key-value store that uses Sinatra. It provides the basic operations GET, SET, DELETE and KEYS.

You can start the application by executing the file app.rb:
```
ruby app.rb
```
Make sure that you have installed the gems sinatra and json.

Usage

Now, let's get our hands dirty. By using PUT and the URI http://localhost:4567/key -d "value" we can set a key-value pair. E.g. we can set the key ruby and the value Is awesome:
```
curl -X PUT http://localhost:4567/ruby -d "Is awesome"
```
In order to get a key's value, we can use GET and the URI http://localhost:4567/key. E.g. we can get the value of ruby:
```
curl http://localhost:4567/ruby
```
If we want to list all keys currently set we can use GET and the URI http://localhost:4567/_/keys:
```
curl http://localhost:4567/_/keys
```
Last but not least: you can delete a key by using DELETE and the URI http://localhost:4567/key. E.g. we can delete the key ruby:
```
curl -X DELETE http://localhost:4567/ruby
```
Future Plans
- Namespaced key-value pairs
app.rb
```
require "sinatra"
require "json"

Dir.mkdir("data") unless File.directory?("data")

get "/:key" do |key|
  if File.exists?("data/#{key}")
    send_file("data/#{key}")
  else
    [404, "Key #{key} does not exist."]
  end
end

get "/_/keys" do
  keys = Dir.entries("data") - [".", ".."]
  keys.to_json
end

put "/:key" do |key|
  data = request.body.read
  File.open("data/#{key}", "w") do |f|
    f.write(data)
  end
  "OK."
end

delete "/:key" do |key|
  if File.exists?("data/#{key}")
    File.delete("data/#{key}")
    "OK."
  else
    [404, "Key #{key} does not exist."]
  end
end
```
View full entry
Finished in 12th place with a final score of 2.9/5. (View the Gist)

Comment

James Urquhart

README.markdown

DumbStore

A rather dumb storage solution. By default all data will be stored in "database.dumb" in the current directory.

Usage

Usage: dumbstore.rb [options] set/get/delete/keys <key> <value>
    -d, --database PATH              Path to database
    -h, --help                       Display this screen

e.g.

Setting, getting and deleting a value

$ dumbstore.rb set answer 42
$ dumbstore.rb get answer # 42
$ dumbstore.rb delete answer
$ dumbstore.rb get answer # error code

Getting keys

$ dumbstore.rb set key1 1
$ dumbstore.rb set key2 2
$ dumbstore.rb keys key* # key1 key2

dumbstore.rb

#!/usr/bin/env ruby
require 'optparse'

OPTS = {}

def error_out(msg)
  puts msg; exit false
end

def entry(args)
  error_out($optusage) if args.length < 2
  cmd, key = args
  dbp = OPTS[:path] || "#{Dir.getwd}/database.dumb"
  
  if !File.directory?(dbp)
    Dir.mkdir(dbp) rescue error_out("Could not load database #{dbp}")
  end
  
  case cmd.downcase.to_sym
  when :set
    error_out("Please specify a value") if args.length < 3
    File.open("#{dbp}/#{key}", 'w'){|f| f.write args[2]} rescue error_out("Couldn't set #{key}")
  when :get
    print File.open("#{dbp}/#{key}").read rescue exit false
  when :keys
    puts Dir["#{dbp}/#{key}"].map{|f|File.basename(f)}.join(' ')
  when :delete
    FileUtils.rm("#{dbp}/#{key}") rescue error_out("Couldn't delete #{key}")
  else
    error_out("Unknown command #{cmd}")
  end
  
  exit true
end

OptionParser.new do |u|
  u.banner = "Usage: dumbstore.rb [options] set/get/delete/keys <key> <value>"
  u.on( '-d', '--database PATH', 'Path to database' ) { |path| OPTS[:path] = path }
  u.on( '-h', '--help', 'Display this screen' ) { puts u; exit }
  $optusage=u
end.parse!

entry(ARGV)

View full entry

Finished in 13th place with a final score of 2.9/5. (View the Gist)

Luke Cowell

fs_store.rb

require 'fileutils'
require 'benchmark'

class FsStore
  def initialize(path)
    self.base = path
  end
  
  def get(key)
    path = path_for_key(key)
    Marshal.load(File.read(path))
  end

  def set(key, value)
    path = path_for_key(key)
    
    begin
      f = File.open(path, "w+")
      f.print Marshal.dump(value)
    ensure
      f.close
    end
    value
  end
  
  def list
    path = File.join(base, "*", "*")
    Dir.glob(path).map do |dir|
      dir.split("/").last
    end
  end
  
  def delete(key)
    path = path_for_key(key)
    File.delete(path)
  end
  
  private

  def path_for_key(key)
    key_string = key.to_s
    [base_path_for_key(key_string), key_string].join("/")
  end

  def base_path_for_key(key)
    key_string = key.to_s
    path = [base, key_string[0].chr].join("/")
    FileUtils.mkdir_p(path)
    path
  end
  
  def base=(path)
    FileUtils.mkdir_p(path)
    @base = path
  end
  
  def base
    @base
  end
end

benchmark.rb

require 'fs_store'

letters = ("a".."z").to_a
count = 10000
keys = []
heading "generating 1000 random keys"
count.times do
  key = ""
  10.times { key << letters[rand(25)] }
  keys << key
end

f = FsStore.new("/tmp/random")

puts start_time = Time.now
keys.each do |key|
  f.set(key, "blah")
end
keys.each do |key|
  f.get(key)
end
puts end_time = Time.now

puts end_time - start_time

runner.rb

require 'fs_store'

def heading(text)
  puts "===#{text}==="
end

heading "initialize the store with a path"
f =  FsStore.new("/tmp/data2")

heading "set some keys"
f.set("hello", "world")
f.set(:foo, "bar")

heading "retrieve those keys with a symbol or string"
puts f.get(:hello)
puts f.get("foo")

heading "listing our objects"
puts f.list

heading "store objects"
f.set("time", Time.now)
puts f.get("time").class

heading "delete keys"
f.delete("hello")

README.textile

This is my modest attempt at a key value store. I use the filesystem to store any object or data requested to be set. `runner` will give you a demonstration of how it works. Right now, the folder splitting depth is 1 so a key like apple would be in /a/apple.

I included a `benchmark` as I’d be curious as to how this entry performs compared to others. Even then my benchmark is far from exhaustive. My keys in the test are random, a better test would see what impact grouping keys together would have. eg (aa, ab, ac). FsStore may also perform well with smaller numbers of keys, but when the number of keys gets to be over a certain size, I can see there being a significant performance hit as the OS tries to find one file in a directory of 10000. I also noticed on my system, which is a mac, that creating 10000 keys at once, makes mds get really busy, I should probably put the data store in a folder with .noindex appended so mds leaves it alone.

View full entry

Finished in 14th place with a final score of 2.9/5. (View the Gist)

Ricardo Mendes

Gemfile

source :rubygems

gem 'sinatra'
gem 'haml'

README.markdown

Massive
Multiplayer
Online
Ephemeral
Key
Value
Storage

MMO

It's meant to be run on a simple Heroku instance, and anyone can pitch in key/value pairs

Ephemeral

Besides the obvious effects of anyone being able to access and modify the data, everything is stored in memory with no hard persistance. This is by design.

So, whenever the instances restarts or goes down, so does everything in it. This is subject to change in the future.

KVS

You store a value associated with a key. That's it.

The End

The app contains instructions on how to move data in and out.

Basically just use HTTP methods and you're golden.

app.rb

require 'sinatra'
require 'haml'
require 'json'

# initial data just to have something to fiddle with
configure do
  set :storage, { 'init' => 'all' , 'dumb' => 'dumber' }
end

get '/' do
  @store = settings.storage

  haml :index
end

get '/keys' do
  settings.storage.keys.join("\n")
end

get '/:key' do
  key = params[:key].strip
  stg = settings.storage

  stg.has_key?(key) ? stg[key] : ""
end

post '/:key' do
  key   = params[:key].strip
  value = request.body.read
  
  value.empty? ? "" : settings.storage[key] = value
end

delete '/:key' do
  key = params[:key].strip
  
  settings.storage.delete(key) if settings.storage.has_key?(key)
end

__END__

@@ index
%html
  %head
    %title Chester :: MMOEKVS

  %body{:style=>"text-align: center;"}
    %h1 Chester, the MMOEKVS
    (Massive Multiplayer Online Ephemeral Key Value Storage)
    %h2 Never Trust The Cloud.
    
    %table{:style=>"background:#DDD; margin:auto;"}
      %tr
        %td
          %strong SET
        %td curl -d "[value]" http://chester.heroku.com/[key]
      %tr
        %td
          %strong GET
        %td curl http://chester.heroku.com/[key]
      %tr
        %td
          %strong KEYS
        %td curl http://chester.heroku.com/keys
      %tr
        %td
          %strong DELETE
        %td curl -X DELETE http://chester.heroku.com/[key
    
    %table{:style => "margin: 2em auto;"}
      %tr
        %td Key
        %td Value
      - @store.each do |k,v|
        %tr
          %td= k
          %td= v

config.ru

require 'rubygems'
require './app.rb'

use Rack::ShowExceptions

run Sinatra::Application

View full entry

Finished in 15th place with a final score of 2.9/5. (View the Gist)

Paul

README.md

Sloth

Sloth is a very lazy, slow key value store. It was created for Code Brawl. Its extremely simple and essentially uses files for storage.

You can do get, set and delete and also retreive a list of keys.

require 'sloth' db = Sloth.new('db_name')

set

db.set('foo',"honkey") db.set('bar','tonk')

get

puts db.get('foo') => 'honkey'

delete

db.delete('foo')

list keys

puts db.keys => ['bar']

min.rb

class Sloth;attr_accessor :d;def initialize(d);@d=d;Dir.mkdir(d) unless File.directory?(d);end;def set(k,v);l=f(k,"w");l.puts v;end;def get(k);return unless exists?(k);l=f(k,"rb");l.read;end;def delete(k);return unless exists?(k);File.delete("#{@d}/#{k}.slo");end;def keys;Dir.entries(@d).map{|x|x[0..-5]};end;def exists?(k);File.exists? "#{@d}/#{k}.slo";end;def f(k,m);File.open("#{@d}/#{k}.slo",m);end;end

sloth.rb

 class Sloth

    attr_accessor :db

    def initialize(db)
      @db = db
      Dir.mkdir(db)  unless File.directory?(db)
    end

    def set(key, value)
      file = f(key, "w")
      file.puts value
      file.close
    end

    def get(key)
      return unless exists?(key)
      file = f(key, "rb")
      return file.read
    end

    def delete(key)
      return unless exists?(key)
      File.delete("#{@db}/#{key}.slo")
    end

    def keys
       Dir.entries(@db).map {|x| x[0..-5]}
    end

    private

      def exists?(key)
        File.exists? "#{@db}/#{key}.slo"
      end

      def f(key, mode)
        File.open("#{@db}/#{key}.slo", mode)
      end


  end

View full entry

Finished in 16th place with a final score of 2.9/5. (View the Gist)

Jimmy Cuadra

key_value.rb

#!/usr/bin/env ruby -w
# encoding: utf-8

require "json"

class KeyValue
  FILE = "#{ENV['HOME']}/.keyvalue"

  def initialize
    begin
      @data = JSON.parse(File.read(FILE))
    rescue Errno::ENOENT, JSON::ParserError
      @data = {}
    end
  end

  def get(key)
    @data[key.to_s]
  end

  def set(key, value)
    @data[key.to_s] = value
    save
    value
  end

  def keys
    @data.keys
  end

  def delete(key)
    value = @data.delete(key.to_s)
    save
    value
  end

  private

  def save
    File.open(FILE, "w") { |f| f.write(JSON.generate(@data)) }
  end
end

if $0 == __FILE__
  method, key, value = *ARGV
  method ||= ""
  kv = KeyValue.new

  result = case method.to_sym
  when :get
    kv.get(key)
  when :set
    kv.set(key, value)
  when :keys
    kv.keys
  when :delete
    kv.delete(key)
  else
    <<-eos
Usage:

./key_value.rb get [key]
./key_value.rb set [key] [value]
./key_value.rb keys
./key_value.rb delete [key]
eos
  end

  puts result if result
end

README.textile

KeyValue

KeyValue is a simple key value store written in Ruby for Codebrawl #9. It provides programmatic and command line access to the store. The data is persisted as JSON, serialized to a file.

Programmatic access:

require "key_value"

kv = KeyValue.new

kv.set("foo", "bar") # => "bar"
kv.get("foo") # => "bar"
kv.keys # => ["foo"]
kv.delete("foo") # => "bar"

CLI access:

$ ./key_value.rb get [key]
$ ./key_value.rb set [key] [value]
$ ./key_value.rb keys
$ ./key_value.rb delete [key]

View full entry

Finished in 17th place with a final score of 2.7/5. (View the Gist)

John E. Vincent

rubykv-data.rb

require 'msgpack'
module RubyKV
  class Data

    attr_reader :data_dir, :name

    def initialize(name, data_dir)
      @name = name
      @data_dir = data_dir
      @store = {}
    end

    def get(key)
      @store[key]
    end

    def set(key, value)
      if @store.has_key?(key) 
        puts "Overwritting existing key: #{key}"
      end
      @store[key] = value
    end

    def delete(key)
      @store.delete key
    end

    def keys
      @store.keys
    end

    def save
      begin
        File.open("#{@data_dir}/#{@name}.mpk", 'w'){|f| f.write(@store.to_msgpack) }
        puts "Database saved!"
      rescue Exception => e
        puts "Error saving database: #{e}"
      end
    end

    def load
      begin
        data = File.open("#{@data_dir}/#{@name}.mpk", 'r')
        @store = MessagePack.unpack data.read
        data.close
        puts "Database loaded"
      rescue Exception => e
        puts "Error loading database: #{e}"
      end
    end
  end
end


# @foo = RubyKV::Data.new "mystore", "/tmp/"
# 1000.times do |n|
#   @foo.set "key_#{n}", "data_#{n}"
# end
# @foo.save

# @foo = RubyKV::Data.new "mystore", "/tmp/"
# @foo
# @foo.load
# @foo

View full entry

Finished in 18th place with a final score of 2.5/5. (View the Gist)

Key/value stores

Prize

This contest is finished

The simplest RESTy database ever

Try it out yourself!

HashProxy

Requirements

Example

Thoughts

Mnemosine

Capabilities

Key

delete_all

delete(k)

keys

exists(k)

randomkey

rename(oldkey, newkey)

renamenx(oldkey, newkey)

select_keys(src = nil)

match_keys(regex, opts = "")

Strings/Integers

set(k, v)

get(k)

append(k, v)

incr(k)

decr(k)

incrby(k, v)

decrby(k, v)

Hashes

hset(k, sk, v)

hsetnx(k, sk, v)

hget(k, sk)

hkeys(k)

hdel(k, sk)

hexists(k, sk)

hlen(k)

hgetall(k)

hincr(k, sk)

hdecr(k, sk)

hincrby(k, sk, v)

hdecrby(k, sk, v)

hmset(k, v)

hmget(k)

Arrays

lset(k, sk, v)

lget(k, sk)

lrange(k, start, stop)

linsert(k, replace, v)

Notes

Requirements

Ruby API

Socket Interface

Geist

Usage

Internals

Caveats

License

Credits

Gingerbread

Requirements

Start a server

Start a Subscriber client to watch for changes

Start a key/value client

A restful KV store

Methods

GET

SET

DELETE

KEYS

Key-Value Store

Usage

README

Usage

Future Plans

DumbStore

Usage

MMO

Ephemeral

KVS