2016年12月5日

Play JSON library

Scala Play 支援自己實作的 play.api.libs.json JSON Lirary

類別

  • JsValue: trait 代表任意一種 JSON value,以下這些是 extend JsValue 的 case classes

  1. JsString
  2. JsNumber
  3. JsBoolean
  4. JsObject
  5. JsArray
  6. JsNull

  • Json 提供 utilties,轉換 JsValue

  • JsPath 代表 JsValue 資料結構中的 path,類似 XML 的 XPath,可使用 pattern 在 JsValue 中進行 data traversal

Converting to a JsValue

  • 以 json String 直接 parsing 並轉換
import play.api.libs.json._

val json: JsValue = Json.parse("""
{
  "name" : "Watership Down",
  "location" : {
    "lat" : 51.235685,
    "long" : -1.309197
  },
  "residents" : [ {
    "name" : "Fiver",
    "age" : 4,
    "role" : null
  }, {
    "name" : "Bigwig",
    "age" : 6,
    "role" : "Owsla"
  } ]
}
""")

直接將上面的 code ,在 activator console 中進行測試,就可以得到測試結果

scala> json
res0: play.api.libs.json.JsValue = {"name":"Watership Down","location":{"lat":51.235685,"long":-1.309197},"residents":[{"name":"Fiver","age":4,"role":null},{"name":"Bigwig","age":6,"role":"Owsla"}]}
  • 以 class 的方式建立 JsValue
import play.api.libs.json._

val json: JsValue = JsObject(Seq(
  "name" -> JsString("Watership Down"),
  "location" -> JsObject(Seq("lat" -> JsNumber(51.235685), "long" -> JsNumber(-1.309197))),
  "residents" -> JsArray(Seq(
    JsObject(Seq(
      "name" -> JsString("Fiver"),
      "age" -> JsNumber(4),
      "role" -> JsNull
    )),
    JsObject(Seq(
      "name" -> JsString("Bigwig"),
      "age" -> JsNumber(6),
      "role" -> JsString("Owsla")
    ))
  ))
))

使用 Writes converters

scala converting to a JsValue 是以 Json.toJsonT(implicit writes: Writes[T]) 進行轉換的,這需要 Writes 物件的幫忙。Play JSON API 提供了一個 implicit Writes 物件,可以轉換 Int, Double, String, Boolean ...,同時也支援 collections

import play.api.libs.json._

// basic types
val jsonString = Json.toJson("Fiver")
val jsonNumber = Json.toJson(4)
val jsonBoolean = Json.toJson(false)

// collections of basic types
val jsonArrayOfInts = Json.toJson(Seq(1, 2, 3, 4))
val jsonArrayOfStrings = Json.toJson(List("Fiver", "Bigwig"))

如果要轉換自己的資料物件類別,就須要提供一個 implicit Writes converter

case class Location(lat: Double, long: Double)
case class Resident(name: String, age: Int, role: Option[String])
case class Place(name: String, location: Location, residents: Seq[Resident])
import play.api.libs.json._

implicit val locationWrites = new Writes[Location] {
  def writes(location: Location) = Json.obj(
    "lat" -> location.lat,
    "long" -> location.long
  )
}

implicit val residentWrites = new Writes[Resident] {
  def writes(resident: Resident) = Json.obj(
    "name" -> resident.name,
    "age" -> resident.age,
    "role" -> resident.role
  )
}

implicit val placeWrites = new Writes[Place] {
  def writes(place: Place) = Json.obj(
    "name" -> place.name,
    "location" -> place.location,
    "residents" -> place.residents)
}

val place = Place(
  "Watership Down",
  Location(51.235685, -1.309197),
  Seq(
    Resident("Fiver", 4, None),
    Resident("Bigwig", 6, Some("Owsla"))
  )
)

val json = Json.toJson(place)

也可以使用 combinator pattern

import play.api.libs.json._
import play.api.libs.functional.syntax._

implicit val locationWrites: Writes[Location] = (
  (JsPath \ "lat").write[Double] and
  (JsPath \ "long").write[Double]
)(unlift(Location.unapply))

implicit val residentWrites: Writes[Resident] = (
  (JsPath \ "name").write[String] and
  (JsPath \ "age").write[Int] and
  (JsPath \ "role").writeNullable[String]
)(unlift(Resident.unapply))

implicit val placeWrites: Writes[Place] = (
  (JsPath \ "name").write[String] and
  (JsPath \ "location").write[Location] and
  (JsPath \ "residents").write[Seq[Resident]]
)(unlift(Place.unapply))

Traversing a JsValue structure

可以 traverse JsValue 並取得特定的資料欄位,這就類似 Scala XML processing

  • Simple Path \
val lat = (json \ "location" \ "lat").get
// returns JsNumber(51.235685)
  • Recursive path \
val names = json \\ "name"
// returns Seq(JsString("Watership Down"), JsString("Fiver"), JsString("Bigwig"))
  • Index lookup (針對 JsArrays)
val bigwig = (json \ "residents")(1)
// returns {"name":"Bigwig","age":6,"role":"Owsla"}

Converting from a JsValue

  • 使用 String utilities
val minifiedString: String = Json.stringify(json)

適合閱讀的格式

val readableString: String = Json.prettyPrint(json)
  • 使用 JsValue.as/asOpt

將 JsValue 轉換為其他資料類別的方式,最簡單的方式就是用 JsValue.asT: T ,但這個方式需要用到一個 implicit converter: Reads[T]

val name = (json \ "name").as[String]
// "Watership Down"

val names = (json \\ "name").map(_.as[String])
// Seq("Watership Down", "Fiver", "Bigwig")

但是 as 可能會產生 JsResultException,所以比較好的方式,應該是改用 JsValue.asOptT: Option[T]

val nameOption = (json \ "name").asOpt[String]
// Some("Watership Down")

val bogusOption = (json \ "bogus").asOpt[String]
// None
  • 使用 validation

會產生兩種 JsResult

  1. JsSuccess: validation/convertion 成功
  2. JsError: validation/conversion 失敗
val json = { ... }

val nameResult: JsResult[String] = (json \ "name").validate[String]

// Pattern matching
nameResult match {
  case s: JsSuccess[String] => println("Name: " + s.get)
  case e: JsError => println("Errors: " + JsError.toJson(e).toString())
}

// Fallback value
val nameOrFallback = nameResult.getOrElse("Undefined")

// map
val nameUpperResult: JsResult[String] = nameResult.map(_.toUpperCase())

// fold
val nameOption: Option[String] = nameResult.fold(
  invalid = {
    fieldErrors => fieldErrors.foreach(x => {
      println("field: " + x._1 + ", errors: " + x._2)
    })
    None
  },
  valid = {
    name => Some(name)
  }
)
  • JsValue 轉換成 data model
case class Location(lat: Double, long: Double)
case class Resident(name: String, age: Int, role: Option[String])
case class Place(name: String, location: Location, residents: Seq[Resident])
import play.api.libs.json._
import play.api.libs.functional.syntax._

implicit val locationReads: Reads[Location] = (
  (JsPath \ "lat").read[Double] and
  (JsPath \ "long").read[Double]
)(Location.apply _)

implicit val residentReads: Reads[Resident] = (
  (JsPath \ "name").read[String] and
  (JsPath \ "age").read[Int] and
  (JsPath \ "role").readNullable[String]
)(Resident.apply _)

implicit val placeReads: Reads[Place] = (
  (JsPath \ "name").read[String] and
  (JsPath \ "location").read[Location] and
  (JsPath \ "residents").read[Seq[Resident]]
)(Place.apply _)


val json = { ... }

val placeResult: JsResult[Place] = json.validate[Place]
// JsSuccess(Place(...),)

val residentResult: JsResult[Resident] = (json \ "residents")(1).validate[Resident]
// JsSuccess(Resident(Bigwig,6,Some(Owsla)),)

Reference

Scala Json in Play

JSON_basics

2016年11月28日

redis for scala play

scala play 預設是使用 ehcache 來當作 cache 的 library,但這個實作常常會在開發的過程當中,會因為程式自動重載的功能,而造成 cache 出現 The play Cache is not alive (STATUS_SHUTDOWN) 的錯誤訊息,在搜尋過後,了解到這可能是 ehcache 的 bug,因此就嘗試將 ehcache 更換成 redis。

安裝 redis

從開發環境到正式環境,需要在不同作業系統安裝 redis,因此我們在這裡記錄不同作業系統安裝 redis 的方法。

windows

在 windows 安裝 [料理佳餚] 在 Windows 上安裝 Redis

Redis on Windows

要注意 Memory Limit 的設定,不然 redis 應該會把記憶體吃光。

用 telnet 或是 redis-cli 可以測試有沒有裝好。

telnet localhost 6379
Mac
sudo port install redis
sudo port load redis

sudo port unload redis

# start redis manually
redis-server /opt/local/etc/redis.conf

要自己去修改 /opt/local/etc/redis.conf

# limited number of clients in developement environment
maxclients 50

# 限制 redis 最多可使用多少記憶體
maxmemory 500MB

# 當記憶體不夠時,要用什麼方式處理
maxmemory-policy volatile-lru
Debian 8

Dotdeb 是Debian 的 3rd party repository,在裡面選擇 Taiwan mirror site。

vi /etc/apt/sources.list.d/dotdeb.list

deb http://ftp.yzu.edu.tw/Linux/dotdeb/ jessie all
deb-src http://ftp.yzu.edu.tw/Linux/dotdeb/ jessie all

安裝 Dotdeb 的 GPG key

wget https://www.dotdeb.org/dotdeb.gpg
sudo apt-key add dotdeb.gpg

安裝 redis server

sudo apt-get update
sudo apt-get install redis-server

啟動/停止

sudo service redis-server start

sudo service redis-server stop

用 redis-benchmark 測試連線狀況

redis-benchmark -q -n 1000 -c 10 -P 5

調整系統參數

sudo sysctl vm.overcommit_memory=1

vi /etc/sysctl.conf
vm.overcommit_memory = 1
Centos 7

安裝 epel

wget -r --no-parent -A 'epel-release-*.rpm' http://dl.fedoraproject.org/pub/epel/7/x86_64/e/

rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-*.rpm

安裝 redis

yum install redis

啟動/停止 redis

systemctl enable redis.service
systemctl start redis.service

systemctl stop redis.service
cluster

redis cluster 是用 sharding 的方式實作的,每一個節點負責一部分 hash slot,一般要建構 cluster 環境,要用到三個 redis node,再加上備援,應該要 6 個 redis node,另外三個是 slave node。

如果要測試細節可以參考這篇文章: Redis Cluster 3.0搭建與使用

或是直接研究官方的文件 Redis cluster tutorial

redis for play

play-redis 是 Play framework 2 cache plugin as an adapter to redis-server,使用這個 framework 的優勢是實作了 play 的 CacheAPI,因此我們可以透過設定,就將 ehcache 調整為 redis。

首先要在 build.sbt 將 disable ehcache,並加上 play-redis 的 libray。

// enable Play cache API (based on your Play version) and optionally exclude EhCache implementation
libraryDependencies += play.sbt.PlayImport.cache exclude("net.sf.ehcache", "ehcache-core")
// include play-redis library
libraryDependencies += "com.github.karelcemus" %% "play-redis" % "1.3.0-M1"

接下來修改 application.conf

# disable default Play framework cache plugin
play.modules.disabled += "play.api.cache.EhCacheModule"

# enable redis cache module
play.modules.enabled += "play.api.cache.redis.RedisCacheModule"

並在 application.conf 中設定 localhost redis 的連線方式

play.cache {
  // redis for Play https://github.com/KarelCemus/play-redis
  redis {
    redis.host="localhost"
    redis.port=6379

    ##redis-server database, 1-15, default:1
    redis.database=1
    ##connection timeout, default:1s (Duration)
    redis.timeout=1s
    ## Akka actor
    redis.dispatcher="akka.actor.default-dispatcher"
    ## Defines which configuration source enable. Accepted values are static, env, custom
    redis.configuration="static"
    ## optional
    #redis.password="null"
    ## optional, Name of the environment variable with the connection string.
    #redis.connection-string-variable=""
    ## Defines behavior when command execution fails.
    #redis.recovery="log-and-default"
  }
}

透過這樣的設定,就可以在不修改 scala 程式的狀況下,就把 ehcache 改成 redis。

2016年11月21日

Play cache API

Scala Play 建議使用 EHCache 作為 cache solution,這在 java 領域是一個常見的套件,目前先看看官方的 cache 方式,畢竟已經整合到 play framework 中,未來再看看有沒有辦法改用 redis。

設定 cache

build/sbt 中增加 cache 這個 libraryDependencies

libraryDependencies ++= Seq(
  cache,
  ...
)

app/controllers/CacheApplication.scala

  • cache.set 新增 cache item
  • cache.get 取得 cache item
  • cache.remove 移除 cache item
  • cache.getOrElse 取得 cache item,如果找不到就新增
package controllers

import java.util.concurrent.TimeoutException
import javax.inject.Inject

import akka.actor.ActorSystem
import akka.pattern.after
import models.{Project, ProjectRepo, TaskRepo}
import play.api.Logger
import play.api.cache.CacheApi
import play.api.libs.concurrent.Execution.Implicits.defaultContext
import play.api.mvc.{Action, Controller}

import scala.concurrent.Future
import scala.concurrent.duration._

class CacheApplication @Inject()(cache: CacheApi)
                           extends Controller {
  def newCache(name: String) = Action {
    val result = s"Add new project ${name} to cache..\n"

    val project: Project = Project(999, name)
    cache.set("project."+name, project)
    Ok(result)
  }

  def newCache2(name: String) = Action {
    val result = s"Add new project ${name} to cache with 5 minuts..\n"

    val project: Project = Project(888, name)
    cache.set("project."+name, project, 5.minutes)

    // 移除 cache item
    //cache.remove("project."+name)
    Ok(result)
  }

  def getCache(name:String) = Action {
    val project: Option[Project] = cache.get[Project]("project."+name)

    val result = project match {
      // 以 Some 測試 name 有沒有存在
      case Some(pj) => s"get project from cache: ${pj.name}"
      case None => s"can't get project from cache.."
    }

    Ok(result)
  }

  def getCache2(name:String) = Action {
    val project: Project = cache.getOrElse[Project]("project."+name) {
      // 如果 cache 中找不到 project.name 就產生一個新的,存到 cache 中
      Project(777, name)
    }

    val result = s"getOrElse project from cache: ${project.name}"
    Ok(result)
  }

}

conf/routes

GET           /cache/:name               controllers.CacheApplication.newCache(name:String)
GET           /cache2/:name               controllers.CacheApplication.newCache2(name:String)
GET           /getcache/:name            controllers.CacheApplication.getCache(name:String)
GET           /getcache2/:name           controllers.CacheApplication.getCache2(name:String)

測試

$ curl 'http://localhost:9000/cache/cproject'
Add new project cproject to cache..

$ curl 'http://localhost:9000/getcache/cproject'
get project from cache: cproject

$ curl 'http://localhost:9000/cache2/cproject'
Add new project cproject to cache with 5 minuts..

$ curl 'http://localhost:9000/getcache2/cproject'
getOrElse project from cache: cproject

預設的 cache store

預設緩存叫 play, 並可以利用 ehcache.xml 來設定新的 cache store。

調整 application.conf

play.cache {
  # If you want to bind several caches, you can bind the individually
  bindCaches = ["db-cache", "user-cache", "session-cache"]
}

在 controller/Application.scala 中使用新的 cache store

import play.api.cache._
import play.api.mvc._
import javax.inject.Inject

class Application @Inject()(
    @NamedCache("session-cache") sessionCache: CacheApi
) extends Controller {

}

Caching HTTP responses

可以將 HTTP response 放進 cache

首先利用 cached: Cached 來 cache actions

import play.api.cache.Cached
import javax.inject.Inject

class Application @Inject() (cached: Cached) extends Controller {

}

例如這個方式,就是將 home 放入 cache

  def cacheAction = cached("homePage") {
    Action {
      Ok("Hello World")
    }
  }

也可以搭配 Authenticated,為每一個 user 暫存不同的 result

def userProfile = Authenticated {
  user =>
    cached(req => "profile." + user) {
      Action {
        Ok(views.html.profile(User.find(user)))
      }
    }
}

可以選擇只要 cache 200 OK 的 response

def get(index: Int) = cached.status(_ => "/resource/"+ index, 200) {
  Action {
    if (index > 0) {
      Ok(Json.obj("id" -> index))
    } else {
      NotFound
    }
  }
}

或是將 404 Not Found 暫存幾分鐘

def get(index: Int) = {
  val caching = cached
    .status(_ => "/resource/"+ index, 200)
    .includeStatus(404, 600)

  caching {
    Action {
      if (index % 2 == 1) {
        Ok(Json.obj("id" -> index))
      } else {
        NotFound
      }
    }
  }
}

Custom Cache API

如果要自己實作新的 Cache API

要先在 application.conf disable EhCacheModule

play.modules.disabled += "play.api.cache.EhCacheModule"

然後用新的 cache API implementation,並 reuse NamedCache 綁定該 implementation。

References

Scala Cache

Play 緩存 API