Key/value stores

This week, the challenge is to write the best key/value storage backend you can think of. It has to be persistent and the goal is for it to be capable of SET, GET, KEYS and DELETE operations, but it’s okay if you don’t have time to implement all of them. We prefer great code over lots of code.

You may implement it as a web service, CLI, Arduino serial adapter… You choose your environment. In the end, I think it would be funny to see lots of funky solutions.

To submit an entry, just put your work in a Gist and include a README to explain what it does and how it works (with code samples!). You have one week to enter, so good luck and have fun!

Prize

Once again, 6sync is offering $25 worth of credit for their hosting platform to this week’s winner!

This contest is finished

Congratulations to this week's winners! The entries and the contestant names are shown below.

  • README.md

    Gittr.rb

    Git as a key-value store! Build with Grit, it supports SET, GET, KEYS, and DELETE operations. In addition, we can also get the change history of key/values.

    And since it's Git, we can easily enhance it to include other awesome Git features such as branches, diffs, reverting, and more!

    Example:

    @store = Gittr.new(:repo => File.expand_path('..', __FILE__))
    @store.clear  # Deletes all keys/values from the store
    
    # SET
    @store['lady'] = "gaga"
    
    # GET
    @store['lady'] #=> "gaga"
    
    # KEYS
    @store.keys  #=> ['lady']
    
    # DELETE
    @store.delete('lady') #=> 'gaga'
    
    # LOG
    @store.log('key')
    
    # Produces:
    [
     {"message"=>"all clear","committer"=>{"name"=>"Matt Sears", "email"=>"matt@mattsears.com"}, "committed_date"=>"..."},
     {"message"=>"set 'lady' ", "committer"=>{"name"=>"Matt Sears", "email"=>"matt@mattsears.com"}, "committed_date"=>"..."}
     {"message"=>"delete 'lady' ", "committer"=>{"name"=>"Matt Sears", "email"=>"matt@mattsears.com"}, "committed_date"=>"..."}
    ]
    
    gittr.rb
    require 'yaml'
    require 'grit'
    
    class Gittr
    
      def initialize(options = {})
        @options = options
        unless ::File.exists?(File.join(path,'.git'))
          Grit::Repo.init(path)
        end
      end
    
      # Add the value to the to the store
      #
      # Example
      #   @store.set('key', 'value')
      #
      # Returns nothing
      def set(key, value)
        save("set '#{key}'") do |index|
          index.add(key_for(key), encode(value))
        end
      end
    
      # Shortcut for #set
      #
      # Example:
      #  @store[key] = 'value'
      #
      def []=(key, value)
        set(key, value)
      end
    
      # Retrieve the value for the given key with a default value
      #
      # Example:
      #  @store.get(key)  #=> value
      #
      # Returns the object found in the repo matching the key
      def get(key, value = nil, *)
        if head && blob = head.commit.tree / key_for(key)
          decode(blob.data)
        end
      end
    
      # Shortcut for #get
      #
      # Example:
      #   @store['key']  #=> value
      #
      def [](key)
        get(key)
      end
    
      # Returns an array of key names contained in store
      #
      # Example:
      #  @store.keys  #=> ['key1', 'key2']
      #
      def keys
        head.commit.tree.contents.map{|blob| deserialize(blob.name) }
      end
    
      # Deletes commits matching the given key
      #
      # Example:
      #  @store.delete('key')
      #
      # Returns nothing
      def delete(key, *)
        self[key].tap do
          save("deleted #{key}") {|index| index.delete(key_for(key)) }
        end
      end
    
      # Deletes all contents of the store
      #
      # Returns nothing
      def clear
        save("all clear") do |index|
          if tree = index.current_tree
            tree.contents.each do |entry|
              index.delete(key_for(entry.name))
            end
          end
        end
      end
    
      # The commit log for the given key
      #
      # Example:
      #  @store.log('key') #=> [{"message"=>"Updated key"...}]
      #
      # Returns Array of commit data
      def log(key)
        git.log(branch, key_for(key)).map{ |commit| commit.to_hash }
      end
    
      # Find the key if exists in the git repo
      #
      # Example:
      #  @store.key? 'key'  #=> true
      #
      # Returns true if found; false if not found
      def key?(key)
        !(head && head.commit.tree / key_for(key)).nil?
      end
    
      private
    
      # Format the given key so that it ensures it's git worthy
      def key_for(key)
        key.is_a?(String) ? key : serialize(key)
      end
    
      # Given the file path, return a new Grit::Repo if found
      def git
        @git ||= Grit::Repo.new(path)
      end
    
      # The git branch to use for this store
      def branch
        @options[:branch] || 'master'
      end
    
      # Checks out the branch on the repo
      def head
        git.get_head(branch)
      end
    
      # Commits the the value into the git repository with the given commit message
      def save(message)
        index = git.index
        if head
          commit = head.commit
          index.current_tree = commit.tree
        end
        yield index
        index.commit(message, :parents => Array(commit), :head => branch) if index.tree.any?
      end
    
      # Converts the value to yaml format
      def encode(value)
        value.to_yaml
      end
    
      # Loads value as a Yaml structure
      def decode(value)
        YAML.load(value)
      end
    
      # Convert value to byte stream. This allows keys to be objects too
      def serialize(value)
        Marshal.dump(value)
      end
    
      # Converts value back to an object.
      def deserialize(value)
        Marshal.restore(value) rescue value
      end
    
      # Given that repo path set in the options, return the expanded file path
      def path(key = '')
        @path ||= File.join(File.expand_path(@options[:repo]), key)
      end
    
    end
    
    gittr_test.rb
    require 'minitest/autorun'
    require File.join(File.dirname(__FILE__), 'gittr.rb')
    
    describe Gittr do
    
      @types = {
        "String" => ["lady", "gaga"],
        "Object" => [{:lady => :gaga}, {:gaga => :ohai}]
      }
    
      before do
        @store = Gittr.new(:repo => File.expand_path('..', __FILE__))
        @store.clear
      end
    
      @types.each do |type, (key, key2)|
    
        it "writes String values to keys" do
          @store[key] = "value"
          @store[key].must_equal "value"
        end
    
        it "reads from keys" do
          @store[key].must_be_nil
        end
    
        it "returns a list of keys" do
          @store[key] = "value"
          @store.keys.must_include(key)
        end
    
        it "guarantees that a different String value is retrieved" do
          value = "value"
          @store[key] = value
          @store[key].wont_be_same_as(value)
        end
    
        it "writes Object values to keys" do
          @store[key] = {:foo => :bar}
          @store[key].must_equal({:foo => :bar})
        end
    
        it "guarantees that a different Object value is retrieved" do
          value = {:foo => :bar}
          @store[key] = value
          @store[key].wont_be_same_as(:foo => :bar)
        end
    
        it "returns false from key? if a key is not available" do
          @store.key?(key).must_equal false
        end
    
        it "returns true from key? if a key is available" do
          @store[key] = "value"
          @store.key?(key).must_equal true
        end
    
        it "removes and return an element with a key from the store via delete if it exists" do
          @store[key] = "value"
          @store.delete(key).must_equal "value"
          @store.key?(key).must_equal false
        end
    
        it "returns nil from delete if an element for a key does not exist" do
          @store.delete(key).must_be_nil
        end
    
        it "removes all keys from the store with clear" do
          @store[key] = "value"
          @store[key2] = "value2"
          @store.clear
          @store.key?(key).wont_equal true
          @store.key?(key2).wont_equal true
        end
    
        it "does not run the block if the #{type} key is available" do
          @store[key] = "value"
          unaltered = "unaltered"
          @store.get(key) { unaltered = "altered" }
          unaltered.must_equal "unaltered"
        end
    
        it "stores #{key} values with #set" do
          @store.set(key, "value")
          @store[key].must_equal "value"
        end
    
        it "returns a list of commit history for the key" do
          @store.log(key).wont_be_empty
        end
      end
    
    end
    
    View full entry
    Finished in 1st place with a final score of 3.9/5. (View the Gist)
  • z.md

    Answers to questions you probably have:

    • The getting of keys are handled automatically because they're in /public.
    • Errors will automatically get a non-200 response.
    • Also, crap like http://localhost:4567/foo/../../ will already be trapped by Rack.
    • Running slow? Run it on a ramdisk (and make a cronjob for rsync for persistence) or buy an SSD.
    0-readme.md

    The simplest RESTy database ever

    Oh yeah. It totally is.

    • Running it

      • Type ruby persistence.rb — this runs on http://localhost:4567 by default.
    • Basic CRUD

      • Adding keys: POST to /your/key/name/here
      • Getting keys: GET /your/key/name/here
      • Deleting keys: DELETE /your/key/name/here
    • Group operations

      • Listing keys: GET /your/key/namespace/ (trailing slash)
      • Deleting groups of keys: DELETE /your/key/namespace/ (trailing slash)

    Try it out yourself!

    require 'rest_client'
    R = RestClient
    
    puts "Adding /people/1          => " + R.post('http://localhost:4567/people/1', name: "Jason", age: "26")
    puts "The contents of /people/1 => " + R.get('http://localhost:4567/people/1')
    puts "Available keys:           => " + R.get('http://localhost:4567/people/')
    puts "Deleting /people/1:       => " + R.delete('http://localhost:4567/people/1')
    puts "Available keys:           => " + R.get('http://localhost:4567/people/')
    
    #### Adding /people/1          => OK
    #### The contents of /people/1 => {"name":"Jason","age":"26"}
    #### Available keys:           => ["/people/1"]
    #### Deleting /people/1:       => OK
    #### Available keys:           => []
    
    persistence.rb
    require 'sinatra'
    require 'json'
    require 'fileutils'
    
    before('*') { |path| @path = File.join(settings.public, path) }
    
    post '*' do  # Create/update data
      params.delete "splat"
      FileUtils.mkdir_p File.dirname(@path)
      File.open(@path, 'w') { |f| f.write params.to_json } && "OK"
    end
    
    get '*/' do  # List keys
      Dir["#{@path}*"].select { |f| File.file? f }.map { |f| f.gsub settings.public, '' }.to_json
    end
    
    delete '*' do  # Delete key(s)
      FileUtils.rm_rf(@path) && "OK"
    end
    
    View full entry
    Finished in 2nd place with a final score of 3.9/5. (View the Gist)
  • Gemfile
    source "http://rubygems.org"
    
    # Specify your gem's dependencies in hash_proxy.gemspec
    gemspec
    
    Rakefile
    require "bundler/gem_tasks"
    
    hash_proxy.gemspec
    # -*- encoding: utf-8 -*-
    $:.push File.expand_path("../lib", __FILE__)
    require "hash_proxy/version"
    
    Gem::Specification.new do |s|
      s.name        = "hash_proxy"
      s.version     = HashProxy::VERSION
      s.authors     = ["Sandro Turriate"]
      s.email       = ["sandro.turriate@gmail.com"]
      s.homepage    = ""
      s.summary     = %q{TODO: Write a gem summary}
      s.description = %q{TODO: Write a gem description}
    
      s.rubyforge_project = "hash_proxy"
    
      s.files         = `git ls-files`.split("\n")
      s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
      s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
      s.require_paths = ["lib"]
    
      # specify any dependencies here; for example:
      s.add_runtime_dependency "zmq"
      s.add_runtime_dependency "consistent_hashr"
      s.add_development_dependency "ruby-debug19"
    end
    
    .gitignore
    *.gem
    .bundle
    Gemfile.lock
    pkg/*
    vendor
    
    hash-proxy
    #!/usr/bin/env ruby
    require 'hash_proxy'
    
    endpoint = ARGV[0] && !ARGV[0].empty? ? ARGV[0] : "tcp://127.0.0.1:6789"
    HashProxy::Proxy.new(endpoint).serve
    
    hash-proxy-node
    #!/usr/bin/env ruby
    require 'hash_proxy'
    
    args = ARGV.dup
    args.unshift('register') if args.empty?
    
    endpoint = args[1] || "tcp://127.0.0.1:6789"
    if args[0] == "register"
      HashProxy::Node.new.register(endpoint)
    elsif args[0] == "serve"
      HashProxy::Node.new(endpoint).serve
    else
      abort("Recognized commands are 'register' and 'serve'")
    end
    
    
    README.markdown

    HashProxy

    A distributed, persistent, key/value store providing language-agnostic storage-engines.

    Requirements

    • Ruby 1.9.2 (Fibers)
    • ZeroMQ
    • ConsistentHashr gem

    Example

    1. Start a node to store data

      bundle exec bin/hash-proxy-node

    2. Start the proxy server

      bundle exec bin/hash-proxy

    3. Connect the client to the proxy

    $ bundle console
    >> c = HashProxy::Client.new
    => #<HashProxy::Client:0x00000100b7ff38 @endpoint="tcp://127.0.0.1:6789", @ctx=#<ZMQ::Context:0x00000100b7fee8>, @socket=#<ZMQ::Socket:0x00000100b7fec0>>
    >> c.get('key')
    => nil
    >> c.set('key', 'value')
    => "value"
    >> c.get('key')
    => "value"
    >> c.list
    => ["key"]
    >> c.delete('key')
    => "value"
    >> c.get('key')
    => nil
    >> c.list
    => []
    
    1. Start a new node

      bundle exec bin/hash-proxy-node

      Close the first node with CTRl-C, then check c.list; the array will be empty

    2. Check the dump (persistence)

      The 'dump' log is written out every second if 1000 entries have been made.

      $ cat dump

    3. The log gets restructured (truncated) every 60 seconds, leaving only the relevant changes.

    4. Restarting the server when the log is present will redistribute the key-value pairs to all of the nodes present. It's always recommended to start the nodes before starting the proxy server. The server reditributes the keys almost immediately, and if a node is missing, the key will be stored on the next available node. When the original node returns, the server will search for the key there on the original node, and won't find the value.

    5. Because zmq is the transport protocol, nodes can be written in any language (with a zmq driver). Check out node.py as an example (python node.py).

    6. Don't need the persistence or distribution? Connect directly to a node. A small benchmark shows it to be twice as fast.

    $ bundle exec bin/hash-proxy-node serve
    $ bundle console
    >> c = HashProxy::Client.new
    >> c.set('foo', 'bar')
    >> c.get('foo')
    

    Thoughts

    Having the nodes, and client all connect to a single process makes distribution very simple. Nodes can be added and removed at will without making changes to the client. The proxy server is the only IP/port that needs to be configured. While configuration is dead-simple, the proxy represents a single point of failure in our system. I've made no provisions for dealing with this problem, though it is solvable. I'm curious to know where the major speed bottle-neck is. Early on, I imagined the cache store (node) would be the bottle-neck, which is why I designed them to be language agnostic. I'd love to see a Ruby node compared to a C node. Though, when I connect directly to a node, skipping the proxy, it's clear that the proxy itself creates some latency, which is disappointing.

    node.py
    import zmq
    import atexit
    import socket
    
    class Node:
      def __init__(self, endpoint=None):
        if endpoint:
          self.endpoint = endpoint
        else:
          self.endpoint = "tcp://127.0.0.1:%s" % self.next_available_port()
        self.ctx = zmq.Context()
        self.socket = self.ctx.socket(zmq.REP)
        self.store = {}
        def close_ctx():
          self.socket.close(); self.ctx.term()
        atexit.register(close_ctx)
    
      def serve(self):
        self.socket.bind(self.endpoint)
        print("Node starting on %s" % self.endpoint)
        while True:
          data = self.socket.recv()
          self.process(data)
    
      def process(self, data):
        properties = data.split(":", 3)
        instruction = properties[0]
        key = properties[1] if len(properties) > 1 else None
        value = properties[2] if len(properties) > 2 else None
        if instruction == "LIST":
          def translate(string):
            return string.replace(",", "%2C")
          keys = ",".join(map(translate, self.store.keys()))
          self.send("ACKLIST", keys)
        elif instruction == "SET":
          v = self.store[key] = value
          self.send("ACKSET", v)
        elif instruction == "GET":
          value = self.store[key] if self.store.has_key(key) else ""
          self.send("ACKGET", value)
        elif instruction == "DELETE":
          value = self.store.pop(key) if self.store.has_key(key) else ""
          self.send("ACKDELETE", value)
    
      def register(self, endpoint):
        client = self.ctx.socket(zmq.REQ)
        client.connect(endpoint)
        self.send("NODE", self.endpoint.replace(":", "%3A"), client)
        client.recv()
        def notify_close():
          self.send("NODEGONE", self.endpoint.replace(":", "%3A"), client)
          client.recv()
          client.close()
        atexit.register(notify_close)
        self.serve()
    
      def next_available_port(self):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.bind(("",0))
        s.listen(1)
        port = s.getsockname()[1]
        s.close()
        return port
    
      def send(self, key, value, socket=None):
        socket = socket if socket else self.socket
        return socket.send("%s:%s" % (key.upper(), value), zmq.NOBLOCK)
    
    if __name__ == '__main__':
      Node().register('tcp://127.0.0.1:6789')
    
    proxy.rb
    module HashProxy
      class Proxy
        def initialize(endpoint)
          @endpoint = endpoint
          @ctx = ZMQ::Context.new
          @socket = @ctx.socket(ZMQ::REP)
          @nodes = {}
          @filename = 'dump'
          @file = File.open(@filename, 'a')
          @buffer = []
          @restructure_persistence_fork = Thread.new {}
          at_exit { @socket.close; @ctx.close; }
        end
    
        def read_fiber
          Fiber.new do
            while true
              if ZMQ.select([@socket], [], [], 0.5)
                data = @socket.recv
                process(data)
              end
              Fiber.yield
            end
          end
        end
    
        def persistence_fiber
          Fiber.new do
            while true
              if @buffer.size >= 1000 && not_restructuring?
                @file.write @buffer.join("\n")
                @file.puts
                @file.fsync
                @buffer.clear
              end
              Fiber.yield
            end
          end
        end
    
        def persistence_restructure_fiber
          Fiber.new do |tick|
            while true
              until tick > 60 && not_restructuring?
                tick += Fiber.yield
              end
              puts "Restructuring '#{@filename}' for greater efficiency."
              @file.fsync
              pid = fork { RestructurePersistence.new(@filename); exit! }
              @restructure_persistence_fork = Process.detach(pid)
              tick = 0
            end
          end
        end
    
        def tick_manager
          @tick_manager ||= TickManager.new
        end
    
        def not_restructuring?
          @restructure_persistence_fork.stop?
        end
    
        def recover_fiber
          Fiber.new do
            while @nodes.empty?
              Fiber.yield
            end
            puts "Attempting to recover from '#{@filename}'."
            File.open(@filename) do |f|
              f.each do |data|
                process(data.strip, true)
              end
            end
          end
        end
    
        def serve
          @socket.bind @endpoint
          puts "Server starting on #{@endpoint}"
          tick_manager.register(persistence_fiber)
          tick_manager.register(persistence_restructure_fiber)
          fibers = [read_fiber, recover_fiber, tick_manager]
          while true
            fibers.each do |fiber|
              if fiber.alive?
                fiber.resume
              else
                fibers.delete(fiber)
              end
            end
          end
        end
    
        def process(data, noreply=false)
          instruction, key, value = data.split(":", 3)
          case instruction
          when "NODE"
            key = URI.unescape(key)
            @nodes[key] = Client.new(key)
            ConsistentHashr.add_server(key, @nodes[key])
            send("ACK")
          when "NODEGONE"
            key = URI.unescape(key)
            @nodes.delete(key)
            ConsistentHashr.remove_server(key)
            send("ACK")
          when "LIST"
            aggregate_list
          when "SET"
            @buffer << data unless noreply
            client = ConsistentHashr.get(key)
            value = client.set(key, value)
            send("ACKSET", value) unless noreply
          when "GET"
            client = ConsistentHashr.get(key)
            send("ACKGET", client.get(key))
          when "DELETE"
            @buffer << data
            client = ConsistentHashr.get(key)
            send("ACKDELETE", client.delete(key))
          end
        end
    
        def aggregate_list
          lists = @nodes.values.map do |client|
            client.list_raw
          end.join(",")
          send("ACKLIST", lists)
        end
    
        private
    
        def send(key, value=nil, socket=@socket)
          socket.send("#{key.to_s.upcase}:#{value}", ZMQ::NOBLOCK)
        end
    
      end
    
      class TickManager
        def initialize
          @last_tick = Time.now
          @subscribers = []
        end
    
        def register(fiber)
          @subscribers << fiber
        end
    
        def fiber
          @fiber ||= Fiber.new do
            while true
              if tick > 1
                @subscribers.each {|f| f.resume(tick) if f.alive?}
                @last_tick = Time.now
              end
              Fiber.yield
            end
          end
        end
    
        def alive?
          fiber.alive?
        end
    
        def resume
          fiber.resume
        end
    
        def tick
          Time.now - @last_tick
        end
      end
    
    end
    
    version.rb
    module HashProxy
      VERSION = "0.0.1"
    end
    
    node.rb
    module HashProxy
      class Node
        def initialize(endpoint=nil)
          if endpoint
            @endpoint = endpoint
          else
            @endpoint = "tcp://127.0.0.1:#{next_available_port}"
          end
          @ctx = ZMQ::Context.new
          @socket = @ctx.socket(ZMQ::REP)
          at_exit { @socket.close; @ctx.close; }
          @store = {}
        end
    
        def serve
          @socket.bind @endpoint
          puts "Node starting on #{@endpoint}"
          while data = @socket.recv
            process(data)
          end
        end
    
        def process(data)
          instruction, key, value = data.split(":", 3)
          case instruction
          when "LIST"
            send("ACKLIST", @store.keys.map {|s| URI.escape(s.to_s, ',')}.join(","))
          when "SET"
            send("ACKSET", @store[key] = value)
          when "GET"
            send("ACKGET", @store[key])
          when "DELETE"
            send("ACKDELETE", @store.delete(key))
          end
        end
    
        def register(endpoint)
          client = @ctx.socket(ZMQ::REQ)
          client.connect(endpoint)
          send("NODE", URI.escape(@endpoint, ":"), client)
          client.recv
          at_exit do
            send("NODEGONE", URI.escape(@endpoint, ":"), client)
            client.recv
            client.close
          end
          serve
        end
    
        private
    
        def next_available_port
          server = TCPServer.new('127.0.0.1', 0)
          @port = server.addr[1]
        ensure
          server.close if server
        end
    
        def send(key, value, socket=@socket)
          socket.send("#{key.to_s.upcase}:#{value}", ZMQ::NOBLOCK)
        end
    
      end
    end
    
    restructure_persistence.rb
    module HashProxy
      class RestructurePersistence
        def initialize(filename)
          @filename = filename
          @store = {}
          read
          write
        end
    
        def read
          File.open(@filename, 'r') do |f|
            f.each do |data|
              instruction, key, value = data.split(":", 3)
              case instruction
              when "SET"
                @store[key] = data
              when "DELETE"
                @store.delete(key)
              end
            end
          end
        end
    
        def write
          File.open(@filename, 'w') do |f|
            f.write @store.values.join
          end
        end
      end
    end
    
    hash_proxy.rb
    require "hash_proxy/version"
    
    module HashProxy
      require 'zmq'
      require 'consistent_hashr'
      require 'fiber'
      require 'socket'
    
      autoload 'Client', 'hash_proxy/client'
      autoload 'Node', 'hash_proxy/node'
      autoload 'Proxy', 'hash_proxy/proxy'
      autoload 'RestructurePersistence', 'hash_proxy/restructure_persistence'
    
      module ServerRemover
        def remove_server(_name)
          @number_of_replicas.times do |t|
            @circle.delete hash_key("#{_name}+#{t}")
          end
        end
      end
      ConsistentHashr.extend ServerRemover
    end
    
    client.rb
    module HashProxy
      class Client
        def initialize(endpoint="tcp://127.0.0.1:6789")
          @endpoint = endpoint
          @ctx = ZMQ::Context.new
          @socket = @ctx.socket(ZMQ::REQ)
          at_exit { @socket.close; @ctx.close; }
          connect
        end
    
        def connect
          @socket.connect(@endpoint)
        end
    
        def list
          @socket.send("LIST:", ZMQ::NOBLOCK)
          l = process(@socket.recv)
          l = l.split(",").map{|s| URI.unescape(s)}
        end
        alias keys list
    
        def list_raw
          @socket.send("LIST:", ZMQ::NOBLOCK)
          process(@socket.recv)
        end
    
        def get(key)
          @socket.send("GET:#{key}", ZMQ::NOBLOCK)
          process(@socket.recv)
        end
        alias [] get
    
        def set(key, value)
          @socket.send("SET:#{key}:#{value}", ZMQ::NOBLOCK)
          process(@socket.recv)
        end
        alias []= set
    
        def delete(key)
          @socket.send("DELETE:#{key}", ZMQ::NOBLOCK)
          process(@socket.recv)
        end
    
        def process(data)
          instruction, value = data.split(":", 2)
          case instruction
          when "ACKLIST", "ACKSET"
            value
          when "ACKGET", "ACKDELETE"
            value unless value.empty?
          else
            raise "Unknown response: #{data}"
          end
        end
      end
    end
    
    .rvmrc
    rvm use 1.9.2
    
    View full entry
    Finished in 3rd place with a final score of 3.8/5. (View the Gist)
  • Guardfile
    guard 'shell' do
      watch(%r{^lib/(.+)\.rb$}) {|m| `ruby test/test_helper.rb` }
      watch(%r{^test/.+\.rb$})  {|m| `ruby test/test_helper.rb` }
    end
    
    README.markdown

    Mnemosine

    Mnemosine is a key/value store written in Ruby. It was created for the CodeBrawl competition located here: http://codebrawl.com/contests/key-value-stores

    You can run the Mnemosine server by running "./mnemosine" from the root directory. There is also a Ruby adapter you can use. The API covers a small amount of what Redis can do, plus a tiny bit it can't.

    Capabilities

    Mnemosine's API is fairly similar to the Redis API, with a few differences. Like Redis, Mnemosine gives you several data types. You can use different functions depending on the data type.

    Key

    Key functions can be used on any value.

    delete_all

    Deletes every key and value pair in the entire database.

    delete(k)

    Deletes the key and its associated value.

    keys

    Lists all keys in the database.

    exists(k)

    Returns true if the key exists, false otherwise.

    randomkey

    Returns a random key from the database.

    rename(oldkey, newkey)

    Moves the value from one key to another, overwriting any data that be have been on the new key.

    renamenx(oldkey, newkey)

    Like rename, but returns an error if the new key already exists.

    select_keys(src = nil)

    Takes a block that is run on each key/value pair. Hopefully this can be expanded into a full map/reduce system at some point.

    match_keys(regex, opts = "")

    Takes a regular expression and returns any keys that match it.

    Strings/Integers

    These operations only work on strings or integers.

    set(k, v)

    Sets the key to the given value.

    get(k)

    Returns the value associated with the key.

    append(k, v)

    Only works with strings. Appends v to the end of the value associated with k.

    incr(k)

    Only works with integers. Adds one to the integer value associated with k.

    decr(k)

    Only works with integers. Subtracts one from the integer value associated with k.

    incrby(k, v)

    Only works with integers. Adds v to the integer value associated with k.

    decrby(k, v)

    Only works with integers. Subtracts v from the integer value associated with k.

    Hashes

    These functions only work with hashes.

    hset(k, sk, v)

    Assigns v to the sub-key sk.

    hsetnx(k, sk, v)

    Like hset, but it will not overwrite an existing key.

    hget(k, sk)

    Returns the value associated with the given key and sub-key.

    hkeys(k)

    Lists all keys in the hash.

    hdel(k, sk)

    Deletes the given sub-key from the hash, returning its value.

    hexists(k, sk)

    Checks to see if the given sub-key exists in the hash.

    hlen(k)

    Returns the number of keys in the hash.

    hgetall(k)

    Returns the entire hash.

    hincr(k, sk)

    Increments the integer stored in sk.

    hdecr(k, sk)

    Decrements the integer stored in sk.

    hincrby(k, sk, v)

    Adds v to the integer stored in sk.

    hdecrby(k, sk, v)

    Subtracts v from the integer stored in sk.

    hmset(k, v)

    Assigns v to the hash. Note that this overwrites everything. It would probably be preferable to merge it.

    hmget(k)

    Seems to be exactly the same as hgetall. I think I intended for this to take an array of sub-keys and just return those.

    Arrays

    This isn't very fleshed out yet. I intended to accomplish more with it, but my cat got very sick and the entire week has been spent dragging the poor guy to the emergency vet or various specialists. As such, there's not much here yet. I'd like to add some more functions, as well as length limited lists, which could be really cool for caching.

    lset(k, sk, v)

    Sets a value at the index specified by sk.

    lget(k, sk)

    Returns the value associated with sk.

    lrange(k, start, stop)

    Returns the values associated with the indexes between (and including) start and stop.

    linsert(k, replace, v)

    Inserts a value at replace, pushing any later keys further back. Redis is a little weird on this one, and I've emulated some of that weirdness. Redis wants you to specify whether you want to insert the value before or after the given key, which seems pointless. You're programmers, and you know how to add or subtract 1 from the index. However, Redis doesn't treat replace as an index, but as a value! That seems strange to me, and I'm not sure I like it. Still, that's how Redis does it, so I've made this function behave mostly the same.

    Notes

    Obviously Ruby is probably not a great choice for creating a database, as one typically wants very high performance. Instead, consider this to be a fun toy. It uses EventMachine on the server and passes JSON via persistent raw TCP connections. It also was also developed TDD style, so it could be good to look at if you're new to testing or Test::Unit.

    I'll probably hack more features onto this as time permits. I'd like to get replication and sharding working, possibly a map/reduce system, and if I'm feeling really ambitious, perhaps some kind of indexed search.

    There's currently no security on the database, and it only runs on localhost. As I said, this is just a fun little project, not a serious DB engine.

    I'm putting this under the MIT license, so feel free to do whatever you want with it. My only request is that you don't use this as the basis for an entry into the upcoming CodeBrawl competition. While that would be hilarious, it would make me a sad panda.

    Mnemosine offers a search system based on code execution. It's kind of like map reduce, but simpler and worse.

    Requirements

    Mnemosine is only compatible with Ruby 1.9. You'll need to install the following gems to use Mnemosine:

    • eventmachine
    • json
    • slop
    • sourcify

    Run the following command to install sourcify:

    gem install ruby_parser file-tail sourcify

    mnemosine
    #!/usr/bin/env ruby
    require 'rubygems'
    require 'slop'
    require File.join(File.dirname(__FILE__), "lib", "server", "mnemosine")
    
    opts = Slop.parse do
      on :p, :port, "Port the server will run on", true, default: 4291, :as => :integer
      on :s, :host, "Host the server will run on", true, default: 'localhost'
      on :h, :help, 'Show this message', :tail => true, do
        puts help
        exit
      end
    end
    
    Mnemosine::Server.new(opts)
    test_helper.rb
    require 'test/unit'
    require 'FileUtils'
    
    class MnemosineTest < Test::Unit::TestCase
      
      # When run in lib mode, the database just runs as a Ruby lib in your process rather
      # than starting EventMachine and taking TCP connections. Lib mode is only included
      # to make testing easier. It doesn't make any sense to use the DB in lib mode for
      # normal processing.
      #
      # In order to use Mnemosine properly, use the client. The client tests are in
      # the test_client.rb file.
      
      def setup
        @numeric_only_message = "That operation is only valid on: Fixnum"
        @string_only_message = "That operation is only valid on: String"
        @string_numeric_only_message = "That operation is only valid on: Fixnum, String"
        @hash_only_message = "That operation is only valid on: Hash"
        @list_only_message = "That operation is only valid on: Array"
      end
      
      def teardown
        @db.delete_all
      end
      
    end
    
    require File.expand_path(File.join File.dirname(__FILE__), 'test_key.rb')
    require File.expand_path(File.join File.dirname(__FILE__), 'test_string.rb')
    require File.expand_path(File.join File.dirname(__FILE__), 'test_hash.rb')
    require File.expand_path(File.join File.dirname(__FILE__), 'test_list.rb')
    require File.expand_path(File.join File.dirname(__FILE__), 'test_persistence.rb')
    
    require File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "server", "mnemosine.rb"))
    
    class ServerTest < MnemosineTest
      
      def setup
        @db = Mnemosine::Server.new(lib: true)
        super
      end
      
      include KeyTest
      include StringTest
      include HashTest
      include ListTest
      include PersistenceTest
    end
    
    require File.expand_path(File.join(File.dirname(__FILE__), "..", "lib", "client", "mnemosine.rb"))
    
    class ClientTest < MnemosineTest
      
      def setup
        @db = Mnemosine::Client.new
        super
      end
      
      include KeyTest
      include StringTest
      include HashTest
      include ListTest
    end
    
    list.rb
    class Mnemosine
      class Server
        
        def lset(k, sk, v)
          ensure_integer_key(sk) || ensure_array(@storage[k], empty: true) || (@storage[k] ||= []; @storage[k][sk] = v)
        end
        
        def lget(k, sk)
          ensure_integer_key(sk) || ensure_array(@storage[k]) || @storage[k][sk]
        end
        
        def lrange(k, start, stop)
          ensure_array(@storage[k]) || @storage[k][start..stop]
        end
        
        def linsert(k, replace, v)
          g = ensure_array(@storage[k])
          return g if g
          replace_idx = nil
          @storage[k].each.with_index do |val, idx|
            if val == replace
              replace_idx = idx
              break
            end
          end
          replace_idx ? @storage[k].insert(replace_idx, v) : {"error" => "Value to replace not found"}
        end
        
      end
    end
    
    test_hash.rb
    module HashTest
      def test_hset
        @db.hset "foo", "bar", "baz"
        assert_equal "baz", @db.hget("foo", "bar")
      end
      
      def test_hset_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hset("foo", "bar", "baz"))
      end
      
      def test_hget_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hget("foo", "bar"))
      end
      
      def test_hkeys
        @db.hset "foo", "bar", "baz"
        @db.hset "foo", "lol", "cat"
        assert_equal ["bar", "lol"], @db.hkeys("foo").sort
      end
      
      def test_hkeys_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hkeys("foo"))
      end
      
      def test_hdel
        @db.hset "foo", "bar", "baz"
        @db.hset "foo", "lol", "cat"
        @db.hdel "foo", "bar"
        assert_equal ["lol"], @db.hkeys("foo")
      end
      
      def test_hdel_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hdel("foo", "bar"))
      end
      
      def test_hexists
        @db.hset "foo", "bar", "baz"
        assert_equal false, @db.hexists("foo", "lol")
        assert_equal true, @db.hexists("foo", "bar")
      end
      
      def test_hexists_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hexists("foo", "bar"))
      end
      
      def test_hlen
        @db.hset "foo", "bar", "baz"
        @db.hset "foo", "lol", "cat"
        assert_equal 2, @db.hlen("foo")
      end
      
      def test_hlen_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hlen("foo"))
      end
      
      def test_hgetall
        @db.hset "foo", "bar", "baz"
        @db.hset "foo", "lol", "cat"
        assert_equal({"bar" => "baz", "lol" => "cat"}, @db.hgetall("foo"))
      end
      
      def test_hgetall_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hgetall("foo"))
      end
      
      def test_hincr
        @db.hset "foo", "bar", 2
        @db.hincr "foo", "bar"
        assert_equal 3, @db.hget("foo", "bar")
      end
      
      def test_hincr_hash_string
        @db.hset "foo", "bar", "baz"
        assert_equal({"error" => @numeric_only_message}, @db.hincr("foo", "bar"))
      end
      
      def test_hincr_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hincr("foo", "bar"))
      end
      
      def test_hdecr
        @db.hset "foo", "bar", 2
        @db.hdecr "foo", "bar"
        assert_equal 1, @db.hget("foo", "bar")
      end
      
      def test_hdecr_hash_string
        @db.hset "foo", "bar", "baz"
        assert_equal({"error" => @numeric_only_message}, @db.hdecr("foo", "bar"))
      end
      
      def test_hdecr_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hdecr("foo", "bar"))
      end
      
      def test_hincrby
        @db.hset "foo", "bar", 2
        @db.hincrby "foo", "bar", 2
        assert_equal 4, @db.hget("foo", "bar")
      end
      
      def test_hincrby_hash_string
        @db.hset "foo", "bar", "baz"
        assert_equal({"error" => @numeric_only_message}, @db.hincrby("foo", "bar", 2))
      end
      
      def test_hincrby_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hincrby("foo", "baz", 2))
      end
      
      def test_hdecrby
        @db.hset "foo", "bar", 2
        @db.hdecrby "foo", "bar", 2
        assert_equal 0, @db.hget("foo", "bar")
      end
      
      def test_hdecrby_hash_string
        @db.hset "foo", "bar", "baz"
        assert_equal({"error" => @numeric_only_message}, @db.hdecrby("foo", "bar", 2))
      end
      
      def test_hdecrby_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hdecrby("foo", "bar", 2))
      end
      
      def test_hmset_and_get
        @db.hmset "foo", {"bar" => "baz", "lol" => "cat"}
        assert_equal({"bar" => "baz", "lol" => "cat"}, @db.hmget("foo"))
      end
      
      def test_hmset_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hmset("foo", {"bar" => "baz", "lol" => "cat"}))
      end
      
      def test_hmget_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hmget("foo"))
      end
      
      def test_hsetnx
        @db.hsetnx("foo", "bar", "baz")
        assert_equal("baz", @db.hget("foo", "bar"))
      end
      
      def test_hsetnx_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @hash_only_message}, @db.hsetnx("foo", "bar", "baz"))
      end
      
      def test_hsetnx_taken
        @db.hset "foo", "bar", "baz"
        assert_equal({"error" => "That key is already assigned"}, @db.hsetnx("foo", "bar", "lol"))
      end
      
    end
    
    test_string.rb
    module StringTest
      def test_set
        @db.set "foo", "bar"
        assert_equal "bar", @db.get("foo")
      end
      
      def test_set_on_hash
        @db.hset "foo", "bar", "baz"
        assert_equal({"error" => @string_numeric_only_message}, @db.set("foo", "bar"))
      end
      
      def test_get_on_hash
        @db.hset "foo", "bar", "baz"
        assert_equal({"error" => @string_numeric_only_message}, @db.get("foo"))
      end
      
      def test_append_empty
        @db.append "foo", "bar"
        assert_equal "bar", @db.get("foo")
      end
      
      def test_append_present
        @db.set "foo", "lol"
        @db.append "foo", "cat"
        assert_equal "lolcat", @db.get("foo")
      end
      
      def test_append_number
        @db.set "foo", 42
        assert_equal({"error" => @string_only_message}, @db.append("foo", "bar"))
      end
      
      def test_incr
        @db.set "foo", 2
        @db.incr "foo"
        assert_equal 3, @db.get("foo")
      end
      
      def test_incr_string
        @db.set "foo", "bar"
        assert_equal({"error" => @numeric_only_message}, @db.incr("foo"))
      end
      
      def test_decr
        @db.set "foo", 2
        @db.decr "foo"
        assert_equal 1, @db.get("foo")
      end
      
      def test_decr_string
        @db.set "foo", "bar"
        assert_equal({"error" => @numeric_only_message}, @db.decr("foo"))
      end
      
      def test_incrby
        @db.set "foo", 2
        @db.incrby "foo", 2
        assert_equal 4, @db.get("foo")
      end
      
      def test_incrby_string
        @db.set "foo", "bar"
        assert_equal({"error" => @numeric_only_message}, @db.incrby("foo", 2))
      end
      
      def test_decrby
        @db.set "foo", 2
        @db.decrby "foo", 2
        assert_equal 0, @db.get("foo")
      end
      
      def test_decrby_string
        @db.set "foo", "bar"
        assert_equal({"error" => @numeric_only_message}, @db.decrby("foo", 2))
      end
      
    end
    
    test_persistence.rb
    module PersistenceTest
      def test_save
        @db.set "foo", "bar"
        loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
        assert_equal true, @db.save(loc)
        assert File.exist?(File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))), "expected file to exist, but it does not"
        FileUtils.rm loc
      end
    
      def test_save_without_location
        @db.set "foo", "bar"
        assert_equal "No location given" , @db.save.values.first
      end
    
      def test_save_already_exists
        @db.set "foo", "bar"
        loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
        @db.save(loc)
        assert_equal "File already exists" , @db.save(loc).values.first
        FileUtils.rm loc
      end
    
      def test_restore
        @db.set "foo", "bar"
        loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
        @db.save(loc)
        @db.delete "foo"
        @db.restore loc
        assert_equal "bar", @db.get("foo")
        FileUtils.rm loc
      end
    
      def test_restore_no_file
        loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
        assert_equal "File does not exist", @db.restore(loc).values.first
      end
    
      def test_save_default_location
        loc = File.expand_path(File.join(File.dirname(__FILE__), "..", "test_db.mns"))
        @db.file_location = loc
        assert_equal true, @db.save
        FileUtils.rm loc
      end
    end
    
    test_key.rb
    module KeyTest
      
      def test_delete_all
        @db.set "foo", "bar"
        @db.delete_all
        assert_nil @db.get("foo")
      end
      
      def test_delete
        @db.set "foo", "bar"
        assert_equal "bar", @db.delete("foo")
        assert_nil @db.get("foo")
      end
      
      def test_keys
        @db.set "foo", "bar"
        @db.set "lol", "cat"
        assert_equal ["foo", "lol"], @db.keys.sort
      end
      
      def test_exists
        @db.set "foo", "bar"
        assert_equal true, @db.exists("foo")
        assert_equal false, @db.exists("lol")
      end
      
      def test_randomkey
        @db.set "foo", "bar"
        @db.set "lol", "cat"
        assert ["foo", "lol"].include?(@db.randomkey), '"foo" or "lol" expected but was'
      end
      
      def test_rename
        @db.set "foo", "bar"
        @db.rename "foo", "baz"
        assert_nil @db.get "foo"
        assert_equal "bar", @db.get("baz")
      end
      
      def test_renamenx
        @db.set "foo", "bar"
        @db.set "baz", "wut"
        assert_equal({"error" => "That key is already assigned"}, @db.renamenx("foo", "baz"))
        assert_equal "bar", @db.get("foo")
        assert_equal "wut", @db.get("baz")
      end
      
      def test_select_keys
        @db.set "foo", 3
        @db.set "bar", 12
        @db.set "baz", 9
        @db.set "lol", 28
        @db.set "cat", 17
        assert_equal ["bar", "lol"], @db.select_keys {|k, v| v % 2 == 0}.sort
      end
      
      def test_match
        @db.set "foO", "bar"
        @db.set "bar", "wut"
        assert_equal ["foO"], @db.match_keys(/(\w)\1/i)
      end
      
    end
    
    persistence.rb
    class Mnemosine
      class Server
        def save(location = nil)
          if location || @file_location
            persist(location || @file_location)
          else
            {"error" => "No location given"}
          end
        end
        
        def save!(location = nil)
          if location || @file_location
            persist!(location || @file_location)
          else
            {"error" => "No location given"}
          end
        end
        
        def restore(location = nil)
          if location || @file_location
            load(location || @file_location)
          else
            {"error" => "No location given"}
          end
        end
        
        def set_location(location)
          @file_location = location
        end
        
        def unset_location
          @file_location = nil
        end
        
        private
        
        def persist(loc)
          if File.exist?(loc)
            {error: "File already exists"}
          else
            persist!(loc)
          end
        end
        
        def persist!(loc)
          File.open(loc, "w") {|f| f.write @storage.to_json}
          true
        end
        
        def load(loc)
          if File.exist? loc
            @storage = JSON.parse(File.read(loc))
          else
            {"error" => "File does not exist"}
          end
        end
        
      end
    end
    
    process.rb
    class Mnemosine
      class Server
        
        attr_accessor :file_location
        
        API_METHODS = %w[set get delete_all delete keys exists randomkey rename renamenx append incr decr incrby decrby
                         select_keys match_keys hset hget hmset hmget hlen hkeys hincr hincrby hdecr hdecrby hgetall
                         hexists hdel hsetnx lset lget linsert lrange]
        
        def initialize(args = {})
          @storage = new_storage
          @port = args[:p] || 4291
          @host = args[:s] || 'localhost'
          unless args[:lib]
            run_loop
          end
        end
        
        def new_storage
          {}
        end
        
        def receive_data(msg)
          return :send_nothing_back if msg == "\n"
          data = JSON.parse(msg)
          k = data.keys.first
          v = data.values.first
          if API_METHODS.include?(k)
            self.send(k, *v)
          else
            {"error" => "#{k} is not an API method"}
          end
        end
        
        def run_loop
          r = Runner
          r.setup(self)
          EventMachine.run do
            EventMachine.start_server @host, @port, r
            puts "Mnemosine listening at #{@host}:#{@port}"
          end
        end
        
        module Runner
          def self.setup(server)
            @@server = server
          end
          
          def receive_data(data)
            puts "Receiving data: #{data.inspect}"
            msg = @@server.receive_data(data)
            unless msg == :send_nothing_back
              puts "Sending msg: #{msg}"
              send_data({val: msg}.to_json + "\n")
              puts "Sent message"
              puts "\n"
            else
              puts "No response"
            end
          end
        end
        
      end
    end
    
    type_check.rb
    class Mnemosine
      class Server
        
        def self.ensure_type(name, classes, opts = {})
          classes = [classes] unless classes.class == Array
          define_method "ensure_#{name}".to_sym, ->(value, args = {}) do
            if !args[:empty] && !value
              return {"error" => "Cannot perform that operation on an empty value"}
            elsif !value
              return nil
            elsif !classes.include?(value.class)
              return {"error" => (opts[:message] || "That operation is only valid on: #{classes.map {|c| c.to_s}.sort.join(", ")}")}
            end
          end
        end
        
        ensure_type :hash, Hash
        ensure_type :string, String
        ensure_type :numeric, Fixnum
        ensure_type :integer_key, Fixnum, :message => "Key must be an integer"
        ensure_type :string_or_numeric, [String, Fixnum]
        ensure_type :array, Array
        
      end
    end
    
    string.rb
    class Mnemosine
      class Server
        
        def set(k, v)
          ensure_string_or_numeric(@storage[k], empty: true) || @storage[k] = v
        end
        
        def get(k)
          ensure_string_or_numeric(@storage[k], empty: true) || @storage[k]
        end
        
        def append(k, v)
          g = ensure_string(@storage[k], empty: true)
          return g if g
          if exists(k)
            @storage[k] += v
          else
            set k, v
          end
        end
        
        def incr(k)
          ensure_numeric(@storage[k]) || @storage[k] += 1
        end
        
        def decr(k)
          ensure_numeric(@storage[k]) || @storage[k] -= 1
        end
        
        def incrby(k, v)
          ensure_numeric(@storage[k]) || @storage[k] += v
        end
        
        def decrby(k, v)
          ensure_numeric(@storage[k]) || @storage[k] -= v
        end
        
      end
    end
    
    key.rb
    class Mnemosine
      class Server
        
        def delete_all
          @storage = new_storage
        end
        
        def delete(k)
          @storage.delete k
        end
        
        def keys
          @storage.keys
        end
        
        def exists(k)
          !!@storage[k]
        end
        
        def randomkey
          @storage.keys[rand * @storage.keys.length]
        end
        
        def rename(old_key, new_key)
          @storage[new_key] = delete(old_key)
        end
        
        def renamenx(old_key, new_key)
          if exists(new_key)
            {"error" => "That key is already assigned"}
          else
            rename old_key, new_key
          end
        end
        
        def select_keys(src = nil)
          if src
            p = eval(src)
            @storage.select(&p).map(&:first)
          elsif block_given?
            @storage.select {|k, v| yield(k, v)}.map(&:first)
          end
        end
        
        def match_keys(regex, opts = "")
          if regex.class == String
            rgx = Regexp.new(regex, opts)
            @storage.keys.select {|k| k =~ rgx}
          else
            @storage.keys.select {|k| k =~ regex}
          end
        end
        
      end
    end
    
    hash.rb
    class Mnemosine
      class Server
        
        def hset(k, sk, v)
          ensure_hash(@storage[k], empty: true) || (@storage[k] ||= {}; @storage[k][sk] = v)
        end
        
        def hget(k, sk)
          ensure_hash(@storage[k], empty: true) || (@storage[k] && @storage[k][sk])
        end
        
        def hkeys(k)
          ensure_hash(@storage[k], empty: true) || (@storage[k].keys)
        end
        
        def hdel(k, sk)
          ensure_hash(@storage[k], empty: true) || (@storage[k].delete sk)
        end
        
        def hexists(k, sk)
          ensure_hash(@storage[k]) || !!@storage[k][sk]
        end
        
        def hlen(k)
          ensure_hash(@storage[k]) || @storage[k].length
        end
        
        def hgetall(k)
          ensure_hash(@storage[k], empty: true) || @storage[k]
        end
        
        def hincr(k, sk)
          ensure_hash(@storage[k]) || ensure_numeric(@storage[k][sk]) || @storage[k][sk] += 1
        end
        
        def hdecr(k, sk)
          ensure_hash(@storage[k]) || ensure_numeric(@storage[k][sk]) || @storage[k][sk] -= 1
        end
        
        def hincrby(k, sk, v)
          ensure_hash(@storage[k]) || ensure_numeric(@storage[k][sk]) || @storage[k][sk] += v
        end
        
        def hdecrby(k, sk, v)
          ensure_hash(@storage[k]) || ensure_numeric(@storage[k][sk]) || @storage[k][sk] -= v
        end
        
        def hmset(k, v)
          ensure_hash(@storage[k], empty: true) || @storage[k] = v
        end
        
        def hmget(k)
          ensure_hash(@storage[k]) || @storage[k]
        end
        
        def hsetnx(k, sk, v)
          if !@storage[k]
            @storage[k] ||= {sk => v}
          else
            g = ensure_hash(@storage[k])
            if g
              g
            elsif !@storage[k][sk]
              @storage[k][sk] = v
            else
              {"error" => "That key is already assigned"}
            end
          end
        end
        
      end
    end
    
    messages.rb
    require 'socket'
    require 'json'
    
    class Mnemosine
      class Client
        
        private
        
        def send_data(msg)
          @socket.puts msg.to_json
          JSON.parse(@socket.readline)["val"]
        end
        
        def self.api_methods(*mthds)
          mthds.each do |mthd|
            define_method mthd, ->(*args) do
              send_data({mthd => args})
            end
          end
        end
        public
        
        def initialize
          @socket = TCPSocket.open('localhost', 4291)
        end
        
        api_methods :set, :get, :delete_all, :delete, :keys, :exists, :rename, :renamenx, :incr,
                    :decr, :incrby, :decrby, :append, :randomkey, :hset, :hget, :hmset, :hmget,
                    :hlen, :hkeys, :hincr, :hincrby, :hdecr, :hdecrby, :hgetall, :hexists, :hdel,
                    :hsetnx, :lset, :lget, :linsert, :lrange
        
        def select_keys(&blk)
          send_data({"select_keys" => blk.to_source})
        end
        
        def match_keys(regex)
          send_data({"match_keys" => [regex.source, regex.options]})
        end
        
      end
    end
    
    mnemosine.rb
    require 'eventmachine'
    require 'json'
    require 'sourcify'
    
    %w[type_check key string hash list persistence process].each do |file|
      require File.expand_path(File.join(File.dirname(__FILE__), "lib", "#{file}.rb"))
    end
    
    test_list.rb
    module ListTest
      
      def test_lset
        @db.lset "foo", 0, "bar"
        assert_equal "bar", @db.lget("foo", 0)
      end
      
      def test_lset_string_key
        assert_equal({"error" => "Key must be an integer"}, @db.lset("foo", "bar", "baz"))
      end
      
      def test_lget_string_key
        assert_equal({"error" => "Key must be an integer"}, @db.lget("foo", "bar"))
      end
      
      def test_lset_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @list_only_message}, @db.lset("foo", 0, "bar"))
      end
      
      def test_lget_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @list_only_message}, @db.lget("foo", 0))
      end
      
      def test_lrange
        @db.lset "foo", 0, "bar"
        @db.lset "foo", 1, "baz"
        assert_equal ["bar", "baz"], @db.lrange("foo", 0, 1)
      end
      
      def test_lrange_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @list_only_message}, @db.lrange("foo", 0, 1))
      end
      
      def test_linsert
        @db.lset "foo", 0, "bar"
        @db.lset "foo", 1, "baz"
        @db.linsert "foo", "baz", "lol"
        assert_equal ["bar", "lol", "baz"], @db.lrange("foo", 0, 2)
      end
      
      def test_linsert_on_string
        @db.set "foo", "bar"
        assert_equal({"error" => @list_only_message}, @db.linsert("foo", "bar", "baz"))
      end
      
      def test_linsert_value_not_found
        @db.lset "foo", 0, "bar"
        assert_equal({"error" => "Value to replace not found"}, @db.linsert("foo", "lol", "cat"))
      end
      
    end
    
    View full entry
    Finished in 4th place with a final score of 3.7/5. (View the Gist)
  • README.md

    A minimalist append-only key/value store written in Ruby.

    Ruby API

    c = Collection.new('example.db')
    
    c.set('hello', 'world')
    c.get('hello') # => "world"
    
    c.delete('hello')
    c.get('hello') # => nil
    
    c.set('foo', {'baz' => 'qux'})
    c.get('foo')['baz'] # => "qux"
    
    c.set('bar', [1,2,3])
    c.get('bar').first # => 1
    
    c.keys # => ["foo", "bar"]
    

    Every update to the collection results in a JSON object being appended to the data file (example.db in the case above). Yes, even on deletes. Indices are kept in memory for speedy reads and the file can periodically be compacted to keep it from getting too unwieldy in size.

    c.compact
    

    Socket Interface

    Included is a simple socket server interface built with EventMachine. Start listening on a given port and writing to the specified data file by running:

    $ ruby server.rb 3456 example.db
    

    Example client usage:

    $ telnet localhost 3456
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    get foo
    {"baz":"qux"}
    delete foo
    OK
    set hello world
    OK
    get hello
    "world"
    keys
    ["bar","hello"]
    compact
    OK
    exit
    
    server.rb
    require 'json'
    
    class Handlers
      class << self
        def handlers
          @handlers ||= {}
        end
        
        def on(command, respond = false, &blk)
          handlers[command.to_s] = Proc.new do |args|
            begin
              result = instance_exec(*args, &blk)
              reply(respond ? result.to_json : 'OK')
            rescue Exception
              reply('ERROR')
            end
          end
        end
        
        def handler(command)
          handlers[command] || Proc.new { reply('Unrecognized command') }
        end
      end
    end
    
    class CollectionHandlers < Handlers
      on(:get, true) { |key| @collection.get(key) }
      on(:set) { |key, *value| @collection.set(key, value.join(' ')) }
      on(:delete) { |key| @collection.delete(key) }
      on(:keys, true) { @collection.keys }
      on(:compact) { @collection.compact }
      on(:exit) { close_connection_after_writing }
    end
    
    module Server
      def initialize(collection)
        @collection = collection
      end
      
      def reply(data)
        send_data("#{data}\n")
      end
      
      def receive_data(data)
        command, *args = data.split(' ')
        handler = CollectionHandlers::handler(command.downcase)
        instance_exec(args, &handler)
      end
    end
    
    if __FILE__ == $0
      $: << File.dirname(__FILE__)
      
      require 'eventmachine'
      require 'store'
      
      EventMachine::run do
        port = ARGV[0]
        collection = Collection.new(ARGV[1])
        EventMachine::start_server('0.0.0.0', port, Server, collection)
        puts "Listening on port #{port}..."
      end
    end
    
    store.rb
    require 'json'
    
    class Storage
      attr_reader :path
      
      def initialize(path)
        @path = path
        File.open(path, 'a')
      end
      
      def read(position, length)
        JSON.parse(IO.read(@path, length, position))
      end
      
      def write(record)
        File.open(@path, 'a') do |f|
          [f.size, f.write("#{record.to_json}\n")]
        end
      end
      
      def each
        File.open(@path) do |f|
          position = 0
          f.each do |line|
            record = JSON.parse(line.chomp)
            length = line.length
            yield(record, position, length)
            position += length
          end
        end
      end
    end
    
    class Collection
      def initialize(path)
        @index = {}
        @storage = Storage.new(path)
        sync
      end
      
      def get(key)
        indexed = @index[key]
        if indexed
          @storage.read(indexed[:position], indexed[:length])['value']
        end
      end
      
      def set(key, value)
        position, length = @storage.write('key' => key, 'value' => value)
        index(key, position, length)
      end
      
      def delete(key)
        if @index[key]
          @storage.write('key' => key, 'deleted' => true)
          @index.delete(key)
        end
      end
      
      def keys
        @index.keys
      end
      
      def sync
        @index.clear
        @storage.each do |record, position, length|
          if record['deleted']
            @index.delete(record['key'])
          else
            index(record['key'], position, length)
          end
        end
      end
      
      def compact
        compacted = Storage.new(@storage.path + '.compact')
        indices = keys.map do |key|
          position, length = compacted.write('key' => key, 'value' => get(key))
          [key, position, length]
        end
        File.rename(compacted.path, @storage.path)
        @index.clear
        indices.each { |i| index(*i) }
      end
      
      private
      
      def index(key, position, length)
        @index[key] = { :position => position, :length => length }
      end
    end
    
    View full entry
    Finished in 5th place with a final score of 3.6/5. (View the Gist)
  • README.md

    Geist

    Geist is a Git-backed key-value store written in Ruby. You may call it "Git Explicit Item Storage Tool" if you really want too.

    Usage

    require 'geist'
    
    g = Geist.new '~/some/storage/path'
    g.set :foo, 'bar'
    g.set :baz, 123
    g.set :name => 'geist', :platform => 'ruby'
    g.delete :baz
    
    g.keys                 #=> ["foo", "name", "platform"]
    g.get :foo             #=> "bar"
    g.get :baz             #=> nil
    g.get :name, :platform #=> ["geist", "ruby"]
    

    Internals

    To be honest, the introduction was an outright fabrication. Geist is just a Ruby API to misuse Git as a simple key-value store. Git itself is a pretty good key-value store used to preserve blob (file), tree (directory), commit and tag objects.

    The Ruby objects to store as values will be marshalled into Git blob objects. These objects are referenced with lightweight Git tags named by the given key.

    Git will not double store duplicate values. Instead, the different key tags will refer the same Git object.

    Caveats

    As Geist uses Git tags as keys, only objects with a working #to_s method can be used as keys. Additionally, based on Git's ref naming rules, Geist rejects keys that can't be used as Git tag names, e.g. containing non-printable characters or backslashes.

    License

    This code is free software; you can redistribute it and/or modify it under the terms of the new BSD License. A copy of this license can be found in the LICENSE file.

    Credits

    • Sebastian Staudt – koraktor(at)gmail.com
    geist.rb
    require 'fileutils'
    require 'open4'
    
    class Geist
    
      def initialize(path)
        @path = File.expand_path path
    
        if !File.exist? @path
          FileUtils.mkdir_p @path
          cmd 'init'
        elsif !cmd('ls-files').success?
          raise "#{@path} is not a Git repository."
        end
      end
    
      def delete(*keys)
        success = true
    
        keys.each do |key|
          if id_for(key).nil?
            success = false
          else
            cmd "tag -d '#{key}'"
          end
        end
    
        success
      end
    
      def get(*keys)
        return nil if keys.empty?
    
        values = []
        keys.each do |key|
          value = nil
          status = cmd "show '#{key}'" do |stdin, stdout|
            if select [stdout]
              blob = stdout.read.strip
              value = Marshal.load blob unless blob.empty?
            end
          end
          values << (status.success? ? value : nil)
        end
    
        keys.size == 1 ? values.first : values
      end
    
      def keys
        keys = nil
        cmd 'tag -l' do |stdin, stdout|
          keys = stdout.lines.to_a.map(&:strip) if select [stdout]
        end
        keys
      end
    
      def set(keys, value = nil)
        keys = { keys => value } unless keys.is_a? Hash
    
        keys.each do |key, val|
          if key.to_s.match /(?:[\W\s^~:?*\[\\]|\.\.|@\{|(?:\/|\.|\.lock)$|^$)/
            warn "Warning: Invalid key '#{key}'"
            return
          end
    
          delete key unless id_for(key).nil?
    
          id = nil
          cmd 'hash-object --stdin -w' do |stdin, stdout|
            stdin.write Marshal.dump(val)
            stdin.close
            id = stdout.read.strip if select [stdout]
          end
    
          cmd "tag -f '#{key}' #{id}"
        end
      end
    
      private
    
      def cmd(git_cmd, &block)
        cmd = "git --git-dir #{@path} #{git_cmd}"
        status = Open4::popen4 cmd do |pid, stdin, stdout, stderr|
          block.call stdin, stdout if block_given?
    
          stdin.close unless stdin.closed?
          stdout.close
          stderr.close
        end
        status
      end
    
      def id_for(key)
        id = nil
        status = cmd "rev-parse '#{key}'" do |stdin, stdout|
          id = stdout.read.strip if select [stdout]
        end
        status.success? ? id : nil
      end
    
    end
    
    View full entry
    Finished in 6th place with a final score of 3.6/5. (View the Gist)
  • README.md

    Gingerbread

    It's like very simple ActiveRecord for Sinatra but uses cookies instead of databases

    Use cookies in Sinatra like a boss!

    Example:

    require './gingerbread'
    require 'sinatra'
    
    class Person < Gingerbread
      attribute :name
    end
    
    get '/create/:name' do
      person = Person.new(self)
      person.name = params[:name]
      person.save
    
      'Created!'
    end
    
    get '/find/:id' do
      person = Person.find(self, params[:id])
    
      "Hi, #{person.name}"
    end
    
    get '/delete/:id' do
      person = Person.find(self, params[:id])
      person.destroy(self)
    
      "#{person.name} has been deleted!"
    end
    
    gingerbread.rb
    class Gingerbread
    
      def initialize(app, attributes = {})
        @@app = app
        @attributes = attributes
      end
    
      def save
        if @@app.request.cookies['gingerbread']
          data = Marshal.load(@@app.request.cookies['gingerbread'])
        else
          data = {}
        end
    
        data[class_name] ||= {}
    
        @attributes[:id] = next_id(data[class_name])
    
        data[class_name][@attributes[:id]] = self
        @@app.response.set_cookie 'gingerbread', :value => Marshal.dump(data), :path => '/'
      end
    
      def destroy(app)
        data = Marshal.load(app.request.cookies['gingerbread'])
        data[class_name].delete(@attributes[:id])
        app.response.set_cookie 'gingerbread', :value => Marshal.dump(data), :path => '/'
      end
    
      class << self
        def attribute(name, value = nil)
          define_method(name) do
            @attributes[name]
          end
    
          define_method("#{name}=") do |value|
            @attributes[name] = value
          end
        end
    
        def find(app, id)
          id = id.to_i
          if app.request.cookies['gingerbread']
            data = Marshal.load(app.request.cookies['gingerbread'])
            data[self.to_s.downcase.to_sym] ? data[self.to_s.downcase.to_sym][id] : nil
          else
            nil
          end
        end
      end
    
      private
        def class_name
          self.class.to_s.downcase.to_sym
        end
    
        def next_id(hash)
          id = hash.map{|a, b| a}.max
          (id || 0) + 1
        end
    end
    
    View full entry
    Finished in 7th place with a final score of 3.4/5. (View the Gist)
  • README.md

    ZeroDB is a simple key/value store using ZeroMQ as the transport. It's pretty basic and synchronous right now. Keys are written to disk as-is.

    Requirements

    • ruby 1.9.2
    • zmq gem - https://github.com/zeromq/rbzmq

      brew install zeromq # 2.1.x
      gem install zmq
      

    Start a server

    ruby server.rb
    

    Start a Subscriber client to watch for changes

    $ irb -r ./sub.rb
    s = Subscriber.new
    s.subscribe 'a'
    s.on_set { |key| puts "#{key} :)" }
    s.on_delete { |key| puts "#{key} :(" }
    s.run
    

    Start a key/value client

    $ irb -r ./client.rb
    c = Client.new
    c.set 'abc', 123
    c.set 'def', 456
    c.get 'abc', 'def'
    c.each { |key| puts key }
    c.delete 'abc'
    c.each { |key| puts key }
    c.delete 'def'
    
    server.rb
    require 'zmq'
    require 'fileutils'
    
    class DB
      def initialize
        @root = File.join(File.dirname(__FILE__), 'data')
        FileUtils.mkdir_p @root
      end
    
      def get(key)
        name = file(key)
        File.exist?(name) ? IO.read(name) : nil
      end
    
      def set(key, value)
        File.open file(key), 'w' do |io|
          io << value
        end
      end
    
      def delete(key)
        FileUtils.rm_rf key
      end
    
      def each(&block)
        dir = Dir.new @root
        dir.each do |entry|
          next if entry =~ /^\./
          block.call entry
        end
      end
    
      def file(key)
        File.join @root, key
      end
    end
    
    class Server
      def initialize(db, context = nil)
        @db = db
        @context = context || ZMQ::Context.new
        @rep = @context.socket ZMQ::REP
        @rep.bind 'tcp://*:5555'
        @pub = @context.socket ZMQ::PUB
        @pub.bind 'tcp://*:5556'
      end
      
      def run
        while msg = @rep.recv
          cmd, *keys = msg.split ' '
    
          next unless valid_keys?(keys)
    
          method = "#{cmd.downcase}_command"
          if respond_to?(method)
            send(method, keys)
          else
            error "Invalid command: #{cmd}"
          end
        end
      rescue Interrupt
        puts "CLOSING"
        @rep.close
      end
    
      def get_command(keys)
        if keys.length > 0
          @rep.send 'OK', ZMQ::SNDMORE
          last = keys.pop
          keys.each do |k|
            @rep.send @db.get(k).to_s, ZMQ::SNDMORE
          end
          @rep.send @db.get(last).to_s
        else
          error "Need at least 1 key"
        end
      end
    
      def set_command(keys)
        if keys.length == 1
          @db.set keys[0], @rep.recv
          @rep.send 'OK'
          @pub.send "#{keys[0]} SET"
        else
          error "Need only 1 key"
        end
      end
    
      def delete_command(keys)
        keys.each do |k|
          @db.delete k
          @pub.send "#{k} DELETE"
        end
    
        @rep.send 'OK'
      end
    
      def keys_command(keys)
        @rep.send 'OK', ZMQ::SNDMORE
        @db.each do |key|
          @rep.send key, ZMQ::SNDMORE
        end
        @rep.send "DONE"
      end
    
      def error(msg)
        @rep.send "ERROR", ZMQ::SNDMORE
        @rep.send msg
      end
    
      def valid_keys?(keys)
        keys.each do |k|
          if k =~ /^[a-z0-9]+$/
            k.downcase!
          else
            error "Invalid key: #{k.inspect}"
            return false
          end
        end
        true 
      end
    end
    
    Server.new(DB.new).run
    
    sub.rb
    require 'zmq'
    
    class Subscriber
      def initialize(context = nil)
        @context = context || ZMQ::Context.new
        @sub = @context.socket ZMQ::SUB
        @sub.connect 'tcp://127.0.0.1:5556'
        @on_set = @on_delete = nil
      end
    
      def subscribe(channel)
        @sub.setsockopt ZMQ::SUBSCRIBE, channel
      end
    
      def unsubscribe(channel)
        @sub.setsockopt ZMQ::UNSUBSCRIBE, channel
      end
    
      def on_set(&block)
        @on_set = block
      end
    
      def on_delete(&block)
        @on_delete = block
      end
    
      def run
        while msg = @sub.recv
          key, cmd = msg.split ' '
          block = case cmd
          when 'SET'    then @on_set
          when 'DELETE' then @on_delete
          end
          block ? block.call(key) : puts(key)
        end
      rescue Interrupt, IRB::Abort
        puts "CLOSING"
        @sub.close
      end
    end
    
    client.rb
    require 'zmq'
    
    class Client
      class ServerError < StandardError
      end
    
      attr_reader :context, :req
      def initialize(context = nil)
        @context = context || ZMQ::Context.new
        @req = @context.socket ZMQ::REQ
        @req.connect 'tcp://127.0.0.1:5555'
      end
    
      def get(*keys)
        @req.send "GET #{keys * " "}"
        error? || begin
          if keys.size == 1
            @req.recv
          else
            keys.map { |k| @req.recv }
          end
        end
      end
    
      def set(key, value)
        @req.send "SET #{key}", ZMQ::SNDMORE
        @req.send value.to_s
        error? || true
      end
    
      def delete(*keys)
        @req.send "DELETE #{keys * " "}"
        error? || true
      end
    
      def each(&block)
        @req.send "KEYS"
        error? || begin
          while key = @req.recv
            if key == "DONE"
              return
            else
              block.call key
            end
          end
        end
      end
    
      def error?
        if @req.recv != 'OK'
          raise ServerError, @req.recv
        end
      end
    
      def close
        @req.close
      end
    end
    
    View full entry
    Finished in 8th place with a final score of 3.2/5. (View the Gist)
  • README.md

    A restful KV store

    It relies on Sinatra to make everything pretty. Persistence is achieved with Marshal.

    Methods

    GET

    curl "http://kvserver.dev/key1"
    #=> value1
    

    SET

    curl -d "value3" "http://kvserver.dev/key3"
    #=> value3
    

    DELETE

    curl -X delete "http://kvserver.dev/key3"
    #=> value3
    

    KEYS

    curl "http://kvserver.dev"
    #=> key1
        key2
        ...
    
    test.rb
    require './store'
    require 'rack/test'
    
    describe KVStore do
      include Rack::Test::Methods
      
      def app; KVStore; end
      
      let(:pairs){ {"test/key" => "test/value", "test/key2" => "test/value2"} }
      
      before(:each){ KVStore::Store.clear; KVStore::Store.merge! pairs }
      
      it 'should get a value' do
        pairs.each_pair do |k, v|
          get("/#{k}"){ |res| res.body.should == v }
        end
      end
      
      it 'should set a value' do
        post "/new/test", "5"
        last_response.body.should == "5"
      end
      
      it 'should remove a key' do
        pairs.each_pair do |k, v|
          delete("/#{k}"){ |res| res.body.should == v }
        end
      end
      
      it 'should get all the keys' do
        get("/"){ |res| res.body.should == pairs.keys.join("\n") }
      end
    end
    
    store.rb
    require 'sinatra/base'
    
    class KVStore < Sinatra::Base
      set :file, "store.msh"
      set :logging, true
    
      configure do
        Store = Marshal.load(File.read settings.file) rescue {}
      end
      
      helpers do
        def save_to_disk
          open(settings.file, "w") do |f|
            f.write Marshal.dump(Store)
          end
        end
      end
    
      before('/*') do
        @key = params[:splat].first
      end
    
      after '/*' do
        # Write to disk only on actual changes
        save_to_disk if env["REQUEST_METHOD"] =~ /post|delete/i and not body.empty?
      end
    
      get('/')     { body Store.keys.join("\n") }              # KEYS
      get('/*')    { body Store[@key] }                        # GET
      post('/*')   { body Store[@key] = params.flatten.first } # SET
      delete('/*') { body Store.delete(@key) }                 # DELETE
    end
    
    KVStore.run! if __FILE__ == $0
    
    View full entry
    Finished in 9th place with a final score of 3.2/5. (View the Gist)
  • README.md

    RuEDB (Ruby Evented Database)

    Mainly based it out of redis. This basically supports basic key/value store that is accessible through through a REST API. It also stores it as yaml and has the option to encrypt the database given a key. (Although that needs manually changing the file atm.)

    Environment:

    • ruby-1.9.2-p290
    • eventmachine
    • yajl
    • gibberish

    Sample Usage:

    # Run Server
    $ ruby server.rb
    Server started, Ruedb version 0.1.0
    The server is now ready to accept connections on port 6187
    
    # In another terminal
    $ curl http://127.0.0.1:6187/values/name
    $ curl http://127.0.0.1:6187/values/name -d "Some Value Here"
    $ curl http://127.0.0.1:6187/values/name
    $ curl http://127.0.0.1:6187/keys
    $ curl -X DELETE http://127.0.0.1:6187/values/name
    
    # I'm unsure though why curl returns "curl: (52) Empty reply from server" 
    # when value is just one word. Value expected returns fine in the rest console I use
    # in Google Chrome
    
    database.rb
    require 'singleton'
    require 'gibberish'
    require 'yajl'
    
    module Ruedb
    class Database
      include Singleton
    
    class << self
      alias_method :connection, :instance
      def use(database = :default, key = nil)
        dump_database! if @database
        @database = get_path(database)
        @key      = key
        @@value_store, @@key_store = decrypt(key, @database)
      end
    
      def decrypt(key, database)
        return [Hash.new, Array.new] unless(File.exist?(database))
        file = File.open(database)
        raw_data = key ? Gibberish::AES.new(key).dec(file.read) : file.read
        file.close
        Yajl::Parser.new.parse(raw_data)
      end
    
      def dump_database!
        string = Yajl::Encoder.encode([@@value_store, @@key_store])
        encrypted_data = @key ? Gibberish::AES.new(@key).enc(string) : string
        file = File.open(@database, 'w+')
        file.write(encrypted_data)
        file.close
      end
    
      def get_path(database)
        return File.join('/', 'tmp', '.ruedb_store') if database.eql?(:default)
        # Check Ruedb::Config
        #return database
      end
    end
    
      def initialize
        # Initialize Values
      end
    
      def key_store  ; @@key_store  ; end
      def value_store; @@value_store; end
    
      def set(key, value)
        key_store.push(key)
        length, library_key, unique_key = setup_values(key)
    
        setup_length(length)
        setup_library_key(length, library_key)
        value_store[length][library_key][unique_key] = value.to_s
      end
    
      def get(key)
        length, library_key, unique_key = setup_values(key)
        value_store[length][library_key][unique_key] || 'Key Not Found'
      rescue
        'Key Not Found'
      end
    
      def delete(key)
        key_store.delete(key)
        length, library_key, unique_key = setup_values(key)
        value = value_store[length][library_key].delete(unique_key)
        value ? {key => value} : 'Key Not Found'
      rescue
        'Key Not Found'
      end
    
      def keys
        key_store
      end
    
      def clear
        # TODO
      end
    
    private
    
      def setup_values(key)
        key, length = key.to_s, key.length
        mid_length  = length / 2
        library_key = generate_library_key(mid_length, key)
        unique_key  = generate_unique_key(mid_length, length, key)
        [length.to_s, library_key.to_s, unique_key.to_s]
      end
    
      def generate_library_key(mid_length, key)
        key.slice(0, mid_length)
      end
    
      def generate_unique_key(mid_length, length, key)
        unique_key  = key.slice(mid_length, length);
      end
    
      def setup_length(length)
        value_store[length] ||= Hash.new
      end
    
      def setup_library_key(length, library_key)
        value_store[length][library_key] ||= Hash.new
      end
    
    end
    end
    
    server.rb
    $:.push File.expand_path("../", __FILE__)
    
    require 'eventmachine'
    require 'socket'
    
    require 'database'
    require 'version'
    
    module Ruedb
    module Server
    
      def post_init
        @port, @ip = Socket.unpack_sockaddr_in(get_peername)
        puts "Client from #{@ip}:#{@port} connected (currently serving #{EM.connection_count} clients)"
      end
    
      def receive_data(data)
        EM.defer(proc {
          match_data = data.match(/^(PUT|GET|POST|DELETE)\s([^\s]*).*\r\n\r\n(.*)/mx)
          raw_data, http_method, path, body = match_data.to_a
          route_path(http_method, path, body)
        }, proc { |result|
          puts "Result: #{result.inspect}"
          send_data(result)
          close_connection_after_writing
        })
      end
    
      def unbind
        puts "Client from #{@ip}:#{@port} disconnected"
      end
    
      def route_path(http_method, path, body)
        case path
        when /^\/values\/([^\/]*)\/?$/
          execute(http_method, $1, body)
        when /^\/keys\/?$/
          Ruedb::Database.connection.keys
        end
      end
    
      def execute(http_method, key, value = nil)
        case http_method
        when 'POST'
          puts "Store value #{value} into key #{key}"
          Ruedb::Database.connection.set(key, value)
        when 'DELETE'
          puts "Delete store with #{key}"
          Ruedb::Database.connection.delete(key)
        when 'GET'
          puts "Get value with key #{key}"
          Ruedb::Database.connection.get(key)
        end
      end
    
    end
    end
    
    begin
      Ruedb::Database.use(:default)
      EventMachine.run do
        port = 6187
        EventMachine::add_periodic_timer(5) do
          Ruedb::Database.dump_database!
        end
        EventMachine.start_server '127.0.0.1', port, Ruedb::Server
        puts "Server started, Ruedb version #{Ruedb::VERSION}"
        puts "The server is now ready to accept connections on port #{port}"
      end
    rescue SystemExit, Interrupt
      puts "# User requested shutdown..."
      puts "* Saving database snapshot before exiting."
      Ruedb::Database.dump_database!
      puts "* DB saved on disk"
      puts "RuEDB is now ready to exit, thanks!"
    rescue Yajl::ParseError
      puts "Invalid Key to Access Database!"
    end
    
    View full entry
    Finished in 10th place with a final score of 3.2/5. (View the Gist)
  • README.md

    Key-Value Store

    Simplest key-value store that is usable (I think). Uses the stdlib PStore for storage which is not suited to storing large pieces of data, but isn't that the point of a key-value store?

    GET / returns a json encoded array of keys (each as strings).

    POST / creates a new key-value pair

    GET /:key returns the value associated with the key

    DELETE /:key deletes the key (and value)

    Usage

    To play around with it I'm using the htty gem (gem install htty).

    $ ruby key_value.rb
    
    $ htty http://0.0.0.0:4567
    http://0.0.0.0:4567/> get
     200  OK -- 4 headers -- 2-character body
    http://0.0.0.0:4567/> body
    []
    http://0.0.0.0:4567/> query-set key value
    http://0.0.0.0:4567/?key=value> post
     200  OK -- 3 headers -- 16-character body
    http://0.0.0.0:4567/?key=value> query-clear
    http://0.0.0.0:4567/> get
     200  OK -- 4 headers -- 7-character body
    http://0.0.0.0:4567/> body
    ["key"]
    http://0.0.0.0:4567/> cd /key
    http://0.0.0.0:4567/key> get
     200  OK -- 4 headers -- 7-character body
    http://0.0.0.0:4567/key> body
    "value"
    http://0.0.0.0:4567/key> delete
     200  OK -- 4 headers -- 7-character body
    http://0.0.0.0:4567/key> get
     200  OK -- 4 headers -- 4-character body
    http://0.0.0.0:4567/key> body
    null
    http://0.0.0.0:4567/key> cd /
    http://0.0.0.0:4567/> get
     200  OK -- 4 headers -- 2-character body
    http://0.0.0.0:4567/> body
    []
    http://0.0.0.0:4567/> 
    
    key_value.rb
    # http://www.ruby-doc.org/stdlib/libdoc/pstore/rdoc/classes/PStore.html
    require 'pstore'
    require 'sinatra'
    require 'json'
    
    PATH  = File.expand_path('~/pstore')
    STORE = PStore.new(PATH)
    
    get '/' do
      content_type :json
      STORE.transaction { STORE.roots.to_json }
    end
    
    post '/' do
      STORE.transaction do
        params.each {|k,v| STORE[k] = v }
      end
    end
    
    get '/:key' do
      content_type :json
      STORE.transaction { STORE[params[:key]].to_json }
    end
    
    delete '/:key' do
      STORE.transaction { STORE.delete(params[:key]).to_json }
    end
    
    key_value_test.rb
    $: << File.dirname(__FILE__)
    
    require 'key_value'
    require 'minitest/autorun'
    require 'rack/test'
    require 'fileutils'
    
    ENV['RACK_ENV'] = 'test'
    
    class KeyValueTest < MiniTest::Unit::TestCase
      include Rack::Test::Methods
      
      def app
        Sinatra::Application
      end
      
      def teardown
        FileUtils.rm(PATH) if File.exist?(PATH)
      end
      
      def test_lists_keys
        get '/'
        assert last_response.ok?
        assert_equal '[]', last_response.body
        
        post '/?key=value'
        get '/'
        assert_equal '["key"]', last_response.body
      end
      
      def test_sets_and_gets_key_values
        post '/?key=value'
        assert last_response.ok?
        
        get '/key'
        assert last_response.ok?
        assert_equal '"value"', last_response.body
      end
      
      def test_deletes_key
        post '/?a=b'
        get '/a'
        assert_equal '"b"', last_response.body
        
        delete '/a'
        assert last_response.ok?
        get '/a'
        assert_equal 'null', last_response.body
      end
    
    end
    
    View full entry
    Finished in 11th place with a final score of 3.0/5. (View the Gist)
  • README.md

    README

    This is a key-value store that uses Sinatra. It provides the basic operations GET, SET, DELETE and KEYS.

    You can start the application by executing the file app.rb:

    ruby app.rb
    

    Make sure that you have installed the gems sinatra and json.

    Usage

    Now, let's get our hands dirty. By using PUT and the URI http://localhost:4567/key -d "value" we can set a key-value pair. E.g. we can set the key ruby and the value Is awesome:

    curl -X PUT http://localhost:4567/ruby -d "Is awesome"
    

    In order to get a key's value, we can use GET and the URI http://localhost:4567/key. E.g. we can get the value of ruby:

    curl http://localhost:4567/ruby
    

    If we want to list all keys currently set we can use GET and the URI http://localhost:4567/_/keys:

    curl http://localhost:4567/_/keys
    

    Last but not least: you can delete a key by using DELETE and the URI http://localhost:4567/key. E.g. we can delete the key ruby:

    curl -X DELETE http://localhost:4567/ruby
    

    Future Plans

    • Namespaced key-value pairs
    app.rb
    require "sinatra"
    require "json"
    
    Dir.mkdir("data") unless File.directory?("data")
    
    get "/:key" do |key|
      if File.exists?("data/#{key}")
        send_file("data/#{key}")
      else
        [404, "Key #{key} does not exist."]
      end
    end
    
    get "/_/keys" do
      keys = Dir.entries("data") - [".", ".."]
      keys.to_json
    end
    
    put "/:key" do |key|
      data = request.body.read
      File.open("data/#{key}", "w") do |f|
        f.write(data)
      end
      "OK."
    end
    
    delete "/:key" do |key|
      if File.exists?("data/#{key}")
        File.delete("data/#{key}")
        "OK."
      else
        [404, "Key #{key} does not exist."]
      end
    end
    
    View full entry
    Finished in 12th place with a final score of 2.9/5. (View the Gist)
  • README.markdown

    DumbStore

    A rather dumb storage solution. By default all data will be stored in "database.dumb" in the current directory.

    Usage

    Usage: dumbstore.rb [options] set/get/delete/keys <key> <value>
        -d, --database PATH              Path to database
        -h, --help                       Display this screen
    

    e.g.

    Setting, getting and deleting a value

    $ dumbstore.rb set answer 42
    $ dumbstore.rb get answer # 42
    $ dumbstore.rb delete answer
    $ dumbstore.rb get answer # error code
    

    Getting keys

    $ dumbstore.rb set key1 1
    $ dumbstore.rb set key2 2
    $ dumbstore.rb keys key* # key1 key2
    
    dumbstore.rb
    #!/usr/bin/env ruby
    require 'optparse'
    
    OPTS = {}
    
    def error_out(msg)
      puts msg; exit false
    end
    
    def entry(args)
      error_out($optusage) if args.length < 2
      cmd, key = args
      dbp = OPTS[:path] || "#{Dir.getwd}/database.dumb"
      
      if !File.directory?(dbp)
        Dir.mkdir(dbp) rescue error_out("Could not load database #{dbp}")
      end
      
      case cmd.downcase.to_sym
      when :set
        error_out("Please specify a value") if args.length < 3
        File.open("#{dbp}/#{key}", 'w'){|f| f.write args[2]} rescue error_out("Couldn't set #{key}")
      when :get
        print File.open("#{dbp}/#{key}").read rescue exit false
      when :keys
        puts Dir["#{dbp}/#{key}"].map{|f|File.basename(f)}.join(' ')
      when :delete
        FileUtils.rm("#{dbp}/#{key}") rescue error_out("Couldn't delete #{key}")
      else
        error_out("Unknown command #{cmd}")
      end
      
      exit true
    end
    
    OptionParser.new do |u|
      u.banner = "Usage: dumbstore.rb [options] set/get/delete/keys <key> <value>"
      u.on( '-d', '--database PATH', 'Path to database' ) { |path| OPTS[:path] = path }
      u.on( '-h', '--help', 'Display this screen' ) { puts u; exit }
      $optusage=u
    end.parse!
    
    entry(ARGV)
    
    View full entry
    Finished in 13th place with a final score of 2.9/5. (View the Gist)
  • fs_store.rb
    require 'fileutils'
    require 'benchmark'
    
    class FsStore
      def initialize(path)
        self.base = path
      end
      
      def get(key)
        path = path_for_key(key)
        Marshal.load(File.read(path))
      end
    
      def set(key, value)
        path = path_for_key(key)
        
        begin
          f = File.open(path, "w+")
          f.print Marshal.dump(value)
        ensure
          f.close
        end
        value
      end
      
      def list
        path = File.join(base, "*", "*")
        Dir.glob(path).map do |dir|
          dir.split("/").last
        end
      end
      
      def delete(key)
        path = path_for_key(key)
        File.delete(path)
      end
      
      private
    
      def path_for_key(key)
        key_string = key.to_s
        [base_path_for_key(key_string), key_string].join("/")
      end
    
      def base_path_for_key(key)
        key_string = key.to_s
        path = [base, key_string[0].chr].join("/")
        FileUtils.mkdir_p(path)
        path
      end
      
      def base=(path)
        FileUtils.mkdir_p(path)
        @base = path
      end
      
      def base
        @base
      end
    end
    
    benchmark.rb
    require 'fs_store'
    
    letters = ("a".."z").to_a
    count = 10000
    keys = []
    heading "generating 1000 random keys"
    count.times do
      key = ""
      10.times { key << letters[rand(25)] }
      keys << key
    end
    
    f = FsStore.new("/tmp/random")
    
    puts start_time = Time.now
    keys.each do |key|
      f.set(key, "blah")
    end
    keys.each do |key|
      f.get(key)
    end
    puts end_time = Time.now
    
    puts end_time - start_time
    
    runner.rb
    require 'fs_store'
    
    def heading(text)
      puts "===#{text}==="
    end
    
    heading "initialize the store with a path"
    f =  FsStore.new("/tmp/data2")
    
    heading "set some keys"
    f.set("hello", "world")
    f.set(:foo, "bar")
    
    heading "retrieve those keys with a symbol or string"
    puts f.get(:hello)
    puts f.get("foo")
    
    heading "listing our objects"
    puts f.list
    
    heading "store objects"
    f.set("time", Time.now)
    puts f.get("time").class
    
    heading "delete keys"
    f.delete("hello")
    
    README.textile

    This is my modest attempt at a key value store. I use the filesystem to store any object or data requested to be set. `runner` will give you a demonstration of how it works. Right now, the folder splitting depth is 1 so a key like apple would be in /a/apple.

    I included a `benchmark` as I’d be curious as to how this entry performs compared to others. Even then my benchmark is far from exhaustive. My keys in the test are random, a better test would see what impact grouping keys together would have. eg (aa, ab, ac). FsStore may also perform well with smaller numbers of keys, but when the number of keys gets to be over a certain size, I can see there being a significant performance hit as the OS tries to find one file in a directory of 10000. I also noticed on my system, which is a mac, that creating 10000 keys at once, makes mds get really busy, I should probably put the data store in a folder with .noindex appended so mds leaves it alone.

    View full entry
    Finished in 14th place with a final score of 2.9/5. (View the Gist)
  • Gemfile
    source :rubygems
    
    gem 'sinatra'
    gem 'haml'
    
    README.markdown

    Massive
    Multiplayer
    Online
    Ephemeral
    Key
    Value
    Storage

    MMO

    It's meant to be run on a simple Heroku instance, and anyone can pitch in key/value pairs

    Ephemeral

    Besides the obvious effects of anyone being able to access and modify the data, everything is stored in memory with no hard persistance. This is by design.

    So, whenever the instances restarts or goes down, so does everything in it. This is subject to change in the future.

    KVS

    You store a value associated with a key. That's it.

    The End

    The app contains instructions on how to move data in and out.

    Basically just use HTTP methods and you're golden.

    app.rb
    require 'sinatra'
    require 'haml'
    require 'json'
    
    # initial data just to have something to fiddle with
    configure do
      set :storage, { 'init' => 'all' , 'dumb' => 'dumber' }
    end
    
    get '/' do
      @store = settings.storage
    
      haml :index
    end
    
    get '/keys' do
      settings.storage.keys.join("\n")
    end
    
    get '/:key' do
      key = params[:key].strip
      stg = settings.storage
    
      stg.has_key?(key) ? stg[key] : ""
    end
    
    post '/:key' do
      key   = params[:key].strip
      value = request.body.read
      
      value.empty? ? "" : settings.storage[key] = value
    end
    
    delete '/:key' do
      key = params[:key].strip
      
      settings.storage.delete(key) if settings.storage.has_key?(key)
    end
    
    __END__
    
    @@ index
    %html
      %head
        %title Chester :: MMOEKVS
    
      %body{:style=>"text-align: center;"}
        %h1 Chester, the MMOEKVS
        (Massive Multiplayer Online Ephemeral Key Value Storage)
        %h2 Never Trust The Cloud.
        
        %table{:style=>"background:#DDD; margin:auto;"}
          %tr
            %td
              %strong SET
            %td curl -d "[value]" http://chester.heroku.com/[key]
          %tr
            %td
              %strong GET
            %td curl http://chester.heroku.com/[key]
          %tr
            %td
              %strong KEYS
            %td curl http://chester.heroku.com/keys
          %tr
            %td
              %strong DELETE
            %td curl -X DELETE http://chester.heroku.com/[key
        
        %table{:style => "margin: 2em auto;"}
          %tr
            %td Key
            %td Value
          - @store.each do |k,v|
            %tr
              %td= k
              %td= v
    
    config.ru
    require 'rubygems'
    require './app.rb'
    
    use Rack::ShowExceptions
    
    run Sinatra::Application
    
    View full entry
    Finished in 15th place with a final score of 2.9/5. (View the Gist)
  • README.md

    Sloth

    Sloth is a very lazy, slow key value store. It was created for Code Brawl. Its extremely simple and essentially uses files for storage.

    You can do get, set and delete and also retreive a list of keys.

    require 'sloth' db = Sloth.new('db_name')

    set

    db.set('foo',"honkey") db.set('bar','tonk')

    get

    puts db.get('foo') => 'honkey'

    delete

    db.delete('foo')

    list keys

    puts db.keys => ['bar']

    min.rb
    class Sloth;attr_accessor :d;def initialize(d);@d=d;Dir.mkdir(d) unless File.directory?(d);end;def set(k,v);l=f(k,"w");l.puts v;end;def get(k);return unless exists?(k);l=f(k,"rb");l.read;end;def delete(k);return unless exists?(k);File.delete("#{@d}/#{k}.slo");end;def keys;Dir.entries(@d).map{|x|x[0..-5]};end;def exists?(k);File.exists? "#{@d}/#{k}.slo";end;def f(k,m);File.open("#{@d}/#{k}.slo",m);end;end
    
    sloth.rb
     class Sloth
    
        attr_accessor :db
    
        def initialize(db)
          @db = db
          Dir.mkdir(db)  unless File.directory?(db)
        end
    
        def set(key, value)
          file = f(key, "w")
          file.puts value
          file.close
        end
    
        def get(key)
          return unless exists?(key)
          file = f(key, "rb")
          return file.read
        end
    
        def delete(key)
          return unless exists?(key)
          File.delete("#{@db}/#{key}.slo")
        end
    
        def keys
           Dir.entries(@db).map {|x| x[0..-5]}
        end
    
        private
    
          def exists?(key)
            File.exists? "#{@db}/#{key}.slo"
          end
    
          def f(key, mode)
            File.open("#{@db}/#{key}.slo", mode)
          end
    
    
      end
    
    View full entry
    Finished in 16th place with a final score of 2.9/5. (View the Gist)
  • key_value.rb
    #!/usr/bin/env ruby -w
    # encoding: utf-8
    
    require "json"
    
    class KeyValue
      FILE = "#{ENV['HOME']}/.keyvalue"
    
      def initialize
        begin
          @data = JSON.parse(File.read(FILE))
        rescue Errno::ENOENT, JSON::ParserError
          @data = {}
        end
      end
    
      def get(key)
        @data[key.to_s]
      end
    
      def set(key, value)
        @data[key.to_s] = value
        save
        value
      end
    
      def keys
        @data.keys
      end
    
      def delete(key)
        value = @data.delete(key.to_s)
        save
        value
      end
    
      private
    
      def save
        File.open(FILE, "w") { |f| f.write(JSON.generate(@data)) }
      end
    end
    
    if $0 == __FILE__
      method, key, value = *ARGV
      method ||= ""
      kv = KeyValue.new
    
      result = case method.to_sym
      when :get
        kv.get(key)
      when :set
        kv.set(key, value)
      when :keys
        kv.keys
      when :delete
        kv.delete(key)
      else
        <<-eos
    Usage:
    
    ./key_value.rb get [key]
    ./key_value.rb set [key] [value]
    ./key_value.rb keys
    ./key_value.rb delete [key]
    eos
      end
    
      puts result if result
    end
    
    README.textile

    KeyValue

    KeyValue is a simple key value store written in Ruby for Codebrawl #9. It provides programmatic and command line access to the store. The data is persisted as JSON, serialized to a file.

    Programmatic access:

    require "key_value"
    
    kv = KeyValue.new
    
    kv.set("foo", "bar") # => "bar"
    kv.get("foo") # => "bar"
    kv.keys # => ["foo"]
    kv.delete("foo") # => "bar"
    

    CLI access:

    $ ./key_value.rb get [key]
    $ ./key_value.rb set [key] [value]
    $ ./key_value.rb keys
    $ ./key_value.rb delete [key]
    
    View full entry
    Finished in 17th place with a final score of 2.7/5. (View the Gist)
  • rubykv-data.rb
    require 'msgpack'
    module RubyKV
      class Data
    
        attr_reader :data_dir, :name
    
        def initialize(name, data_dir)
          @name = name
          @data_dir = data_dir
          @store = {}
        end
    
        def get(key)
          @store[key]
        end
    
        def set(key, value)
          if @store.has_key?(key) 
            puts "Overwritting existing key: #{key}"
          end
          @store[key] = value
        end
    
        def delete(key)
          @store.delete key
        end
    
        def keys
          @store.keys
        end
    
        def save
          begin
            File.open("#{@data_dir}/#{@name}.mpk", 'w'){|f| f.write(@store.to_msgpack) }
            puts "Database saved!"
          rescue Exception => e
            puts "Error saving database: #{e}"
          end
        end
    
        def load
          begin
            data = File.open("#{@data_dir}/#{@name}.mpk", 'r')
            @store = MessagePack.unpack data.read
            data.close
            puts "Database loaded"
          rescue Exception => e
            puts "Error loading database: #{e}"
          end
        end
      end
    end
    
    
    # @foo = RubyKV::Data.new "mystore", "/tmp/"
    # 1000.times do |n|
    #   @foo.set "key_#{n}", "data_#{n}"
    # end
    # @foo.save
    
    # @foo = RubyKV::Data.new "mystore", "/tmp/"
    # @foo
    # @foo.load
    # @foo
    
    View full entry
    Finished in 18th place with a final score of 2.5/5. (View the Gist)