YALTools | Top Page | Case Studies

1. How to use the YALTools...

YALTools provides commands designed to create, read, update and delete documents on CouchDB.

There are some unique tools, and, especially, csv2json is designed to define view and populates some kind of documents from a csv file in command line.

The "csv2json" is not the essential tool, but it is an example this kind of filter command might be productive in a command line environment.

1-1. Create new database

To create a database, pwdb, please following steps.

$ sbin/mkdb pwdb
{"ok":true}

1-2. Display existing databases

$ sbin/lsdbs
{"database":["_users","pwdb"]}

$ sbin/rmdb pwdb
{"ok":true}
$ sbin/mkdb pwdb
{"ok":true}
$ bin/csv2json -t '["name","pw","uid","gid","gecos","home","shell"]' -s ':' -n  /etc/passwd | \
    bin/postdocs pwdb
$ bin/lsdocs pwdb | \
    head -1 | \
    bin/jsonfmt
{
  "_id":  "a1ff622c5b43d9cf8aba1fa1e20074b2",
  "_rev":  "1-a8d09a9a7c1b4b7d9dfe6af45abef79c",
  "name":  "root",
  "pw":  "x",
  "uid":  "0",
  "gid":  "0",
  "gecos":  "root",
  "home":  "/root",
  "shell":  "/bin/bash"
}

1-4. Define new views

Here is the view definition for pwdb defined in previous section, Populate test data.

"views","uid","map","views","uid","reduce","views","name","map","views","name","reduce"
,,"function(doc) {
  if(doc.uid) {
    emit(doc.uid, 1);
  }
}",,,"_count",,,"function(doc) {
  if(doc.name) {
    emit(doc.name, 1);
  }
}",,,"_count"

A csv file is portable and editable by a text editor or a spreadsheet software, such as emacs and LibreOffice Calc.

These two view definition are placed side-by-side in a horizontal direction. 
This is caused by requirements of "csv2json".

Using YALTools, sotre this csv file in to the pwdb.

$ bin/csv2json samples.csv/pwdb_views.csv | \
    bin/putdesign pwdb test

The registered view definitions can be verified to see the /pwdb/_design/test document.

$ bin/lsdoc /pwdb/_design/test | \
    bin/jsonfmt
{
  "_id":  "_design/test",
  "_rev":  "1-39d91a281979eb7fc62b3f1d4a9f47ab",
  "views":  {
    "uid":    {
      "map":      "function(doc) {
  if(doc.uid) {
    emit(doc.uid, 1);
  }
}",
      "reduce":      "_count"
    },
    "name":    {
      "map":      "function(doc) {
  if(doc.name) {
    emit(doc.name, 1);
  }
}",
      "reduce":      "_count"
    }
  }
}

1-5. Search views by keys

Let's search the views defined in previous section with "lsviews".

Example searchs the all documents sorted by the "name" field.

$ bin/lsviews pwdb test name

Example searchs the document named postfix.

$ bin/lsviews pwdb test name -k '"postfix"'
{"_id":"ed5b14905389ef0d464c2c8509012eeb","_rev":"1-63f1a7b49dbd18b79a1ff59a38bd4410","name":"postfix","pw":"x","uid":"119","gid":"131","gecos":"","home":"/var/spool/postfix","shell":"/bin/false"}

Example searchs documents, each name is beging from "post".

$ bin/lsviews pwdb test name -s '"post"' -e '"postZ"'
{"_id":"ed5b1...","_rev":"1-63f1a7b...","name":"postfix","pw":"x","uid":"119","gid":"131","gecos":"","home":"/var/spool/postfix","shell":"/bin/false"}
{"_id":"ed5b1...","_rev":"1-5c7f03c...","name":"postgres","pw":"x","uid":"113","gid":"126","gecos":"PostgreSQL administrator,,,","home":"/var/lib/postgresql","shell":"/bin/bash"}

1-6. Delete all documents

Here is an example to delete all documents on pwdb database.

$ bin/lsdocs pwdb | \
    bin/chjson '["_deleted",true]' | \
    bin/postdocs pwdb
{"failed_list":[]}
$ bin/lsdocs pwdb
$

2. Working with JSON streams

YALTools provides some tiny stream filters, attachjson ,chjson, grepjson and reducejson. These tools transform an input JSON stream, a file or input from a pipeline.

A JSON stream consists of lines, each line is a complete JSON format string. Here is an example of the JSON stream.

{"_id":"daemon","pw":"x","uid":"1"}
{"_id":"bin","pw":"x","uid":"2"}

This is an text stream, so you can use an usual strem filter like grep and/or sed.

2-1. Creating a JSON stream

YALTools provides lsdocs and lsviews to create json streams.

$ lsdocs pwdb

3. Working with Attachments

3-1. Adding an attachment to the existing design document.

Before executing following commands, please setup the pwdb described at Populate test data.

Now, we have the following design document which was created at the previous part. There is no attachment file.

$ bin/lsdoc /pwdb/_design/test
{}

Let's try to add an attachment file to this design document.

$ bin/lsdoc /pwdb/_design/test | \
    bin/attachjson '["text/plain","passwd.txt","/etc/passwd"]' | \
    bin/putdesign pwdb test

You can simply access the URL; http://127.0.0.1:5984/pwdb/_design/test/passwd.txt

These host and port names depend on your configurations.

3-2. Viewing documents with attachments

The lsdocs command has the "-a" option which shows documents with attachments.

$ bin/lsdocs -a pwdb | grep _attachments | bin/jsonfmt
{
  "_id":  "_design/test",
  "_rev":  "1-88725875de4f34194af1000a1db039ea",
  "_attachments":  {
      "passwd.txt":    {
          "content_type":      "text/plain",
          "revpos":      "1",
          "data":      "cm9vdDp ..."
    }
  }
}

If the "-a" option is omitted, the result is following.

$ bin/lsdocs pwdb | grep _attachments | bin/jsonfmt
{
  "_id":  "_design/test",
  "_rev":  "1-88725875de4f34194af1000a1db039ea",
  "_attachments":  {
      "passwd.txt":    {
          "content_type":      "text/plain",
          "revpos":      "1",
          "length":      "2847",
          "stub":      "true"
    }
  }
}

Even this kind of document can modify other attributes, but cannot PUT or POST to another database.

$ bin/lsdocs pwdb | grep _attachments | bin/chjson '["name","foo"]' | bin/postdocs pwdb
$ bin/lsdocs pwdb | grep _attachments | bin/jsonfmt
{
  "_id":  "_design/test",
  "_rev":  "2-fadd27ed62615522056ade3fad5b23f3",
  "name":  "foo",
  "_attachments":  {
      "passwd.txt":    {
          "content_type":      "text/plain",
          "revpos":      "1",
          "length":      "2847",
          "stub":      "true"
    }
  }
}

3-3. Deleting an attachment

In the example of the previous chapter, we can deletes all attachments in the following way.

$ bin/lsdoc /pwdb/_design/test | \
    bin/reducejson -e '["_attachments"]' | \
    bin/putdesign pwdb test

If there are multiple attachments, it is impossible to delete one of them by this example.

3-3-1. Deleting an attachment from multiple sources

Before executing following commands, please setup the pwdb described at Populate test data.

The previous example is useless to delete an attachment from multiple attachments.

First, we prepares two attached documents on the /pwdb/_design/test document.

$ bin/lsdoc /pwdb/_design/test | \
    bin/attachjson '["text/plain","passwd.txt","/etc/passwd"]' '["text/plain","group.txt","/etc/group"]' | \
    bin/putdesign pwdb test

$ bin/lsdoc /pwdb/_design/test | bin/jsonfmt
{
  "_id":  "_design/test",
  "_rev":  "1-39b847cf4968c12faf40a4ee2774a764",
  "_attachments":  {
    "passwd.txt":    {
      "content_type":      "text/plain",
      "revpos":      "7",
      "length":      "2847",
      "stub":      "true"
    },
    "group.txt":    {
      "content_type":      "text/plain",
      "revpos":      "9",
      "length":      "1295",
      "stub":      "true"
    }
  }
}

To remove the "group.txt" file from attachments, the editjson command can be used.

$ bin/lsdoc /pwdb/_design/test | \
    bin/editjson 'json["_attachments"].delete("group.txt")' | \
    bin/putdesign pwdb test

Let's check the current status.

$ bin/lsdocs pwdb | grep _design/test | bin/jsonfmt
{
  "_id":  "_design/test",
  "_rev":  "10-0bf6fbca0d1b7141ccd469b27f5c0318",
  "_attachments":  {
    "passwd.txt":    {
      "content_type":      "text/plain",
      "revpos":      "7",
      "length":      "2847",
      "stub":      "true"
    }
  }
}

We can also use other general lightweight language like python, instead of the home-brewed editjson command.

First, it adds the deleted "group.txt" again, then removes the "group.txt" file.

$ bin/lsdoc /pwdb/_design/test | \
    bin/attachjson '["text/plain","passwd.txt","/etc/passwd"]' '["text/plain","group.txt","/etc/group"]' | bin/putdesign pwdb test
$ bin/lsdocs pwdb | grep _design/test | bin/jsonfmt
{
  "_id":  "_design/test",
  "_rev":  "10-0bf6fbca0d1b7141ccd469b27f5c0318",
  "_attachments":  {
    "passwd.txt":    {
      "content_type":      "text/plain",
      "revpos":      "7",
      "length":      "2847",
      "stub":      "true"
    }
  }

$ bin/lsdoc /pwdb/_design/test | \
    python -c "import sys;import json; l = sys.stdin.read(); j = json.loads(l); del j['_attachments']['group.txt']; print json.dumps(j)" | \
    bin/putdesign pwdb test
$ bin/lsdoc /pwdb/_design/test | bin/jsonfmt 
{
  "_id":  "_design/test",
  "_rev":  "8-9ec47ef62aab724c50da3a336b5b1cb1",
  "_attachments":  {
      "passwd.txt":    {
          "content_type":      "text/plain",
          "revpos":      "7",
          "length":      "2847",
          "stub":      "true"
    }
  }
}

4. Sample Application - Japan National Postal Code Search Engine

It is the almost same application which is available at the https://www.yadiary.net/postal/main.fcgi, the Japan national post code search application.

I prepared the subset of the code to be easy to run on your environment.

Please refer the following site to get more details; https://github.com/YasuhiroABE/postal-code-searcher

If you feel difficulty to run the application, please keep in touch by e-mail. I will consider the way to distribute my VMware image.