2016/12/26

Scala Play Application for Production

開發時 activator run 啟動的是 DEV mode,會一直檢查程式有沒有修改過,如果有修改,就會自動編譯並 reload,但這個功能會增加 overhead,在 PROD mode 就不需要了。另外 PROD 環境產生的 error page,也不需要像 DEV mode 一樣有太多錯誤的細節。

Production 相關設定

Play 需要一個 secret key 用來對 session cookie 簽章,還有內建的加密 utilities。

application.conf

play.crypto.secret = "newsecret"

該密碼預設為 "changeme",如果沒有修改這個密碼,在 PROD mode 就會發生錯誤,但在 DEV mode 就沒有檢查。


如果想要在 DEV 跟 PROD mode 使用不同的設定檔,可以在 conf 目錄中增加一個 prod-application.conf 新的設定檔,在啟動 PROD mode 時,加上設定檔的附加參數。

準備一個新的 prod-application.conf,內容為

include "application.conf"

play.crypto.secret = "newsecret"

另外同時準備一個 prod-logback.xml,新的 logback 設定檔,將 slick db sql statement 的 log 隱藏掉。


修改 build.sbt ,增加 JavaServerAppPackaging plugin,以及一些 production package 的設定

lazy val root = (project in file(".")).enablePlugins(PlayScala, JavaServerAppPackaging)

// production settings
maintainer := "charley <charley@maxkit.com.tw>"
packageSummary := "Play Slick Sample"
packageDescription := """A fun package description of our software,
  with multiple lines."""

// RPM SETTINGS
rpmVendor := "maxkit"
//rpmLicense := Some("BSD")
rpmChangelogFile := Some("changelog.txt")

Play 預設是使用 Netty,Netty 提供了一些參數可以調整,我們可以在 application.conf 中,調整這些參數。

play.server {

  # The server provider class name
  provider = "play.core.server.NettyServerProvider"

  netty {

    # The number of event loop threads. 0 means let Netty decide, which by default will select 2 times the number of
    # available processors.
    eventLoopThreads = 0

    # The maximum length of the initial line. This effectively restricts the maximum length of a URL that the server will
    # accept, the initial line consists of the method (3-7 characters), the URL, and the HTTP version (8 characters),
    # including typical whitespace, the maximum URL length will be this number - 18.
    maxInitialLineLength = 4096

    # The maximum length of the HTTP headers. The most common effect of this is a restriction in cookie length, including
    # number of cookies and size of cookie values.
    maxHeaderSize = 8192

    # The maximum length of body bytes that Netty will read into memory at a time.
    # This is used in many ways.  Note that this setting has no relation to HTTP chunked transfer encoding - Netty will
    # read "chunks", that is, byte buffers worth of content at a time and pass it to Play, regardless of whether the body
    # is using HTTP chunked transfer encoding.  A single HTTP chunk could span multiple Netty chunks if it exceeds this.
    # A body that is not HTTP chunked will span multiple Netty chunks if it exceeds this or if no content length is
    # specified. This only controls the maximum length of the Netty chunk byte buffers.
    maxChunkSize = 8192

    # Whether the Netty wire should be logged
    log.wire = false

    # The transport to use, either jdk or native.
    # Native socket transport has higher performance and produces less garbage but are only available on linux 
    transport = "jdk"

    # Netty options. Possible keys here are defined by:
    #
    # http://netty.io/4.0/api/io/netty/channel/ChannelOption.html
    #
    # Options that pertain to the listening server socket are defined at the top level, options for the sockets associated
    # with received client connections are prefixed with child.*
    option {

      # Set the size of the backlog of TCP connections.  The default and exact meaning of this parameter is JDK specific.
      # SO_BACKLOG = 100

      child {
        # Set whether connections should use TCP keep alive
        # SO_KEEPALIVE = false

        # Set whether the TCP no delay flag is set
        # TCP_NODELAY = false
      }

    }

  }
}

activator 常用指令

ex: activator clean

  • clean: 刪除 target 目錄中的編譯結果
  • update: 下載的 libraries
  • compile: 編譯專案,產生到 target 目錄中
  • eclipse: 產生 eclipse 專案 project file,使用前要先執行 compile
  • run: 啟動 project,也可以用 activator "run 8888" 啟動到 TCP Port 8888
  • publish: 產生專案的 jar,發佈到 build.sbt 中設定的 repository
  • publish-local: 發佈到本機的 repository/local 目錄
  • stage: 產生 production 專案的 script,通常放在 target/universal/stage 目錄中

產生 stage project

activator stage

完整地產生新的 stage project 要執行

activator clean update compile stage

產生 production project tar.gz file

activator universal:packageZipTarball

PROD mode

以下兩種方式可以啟動 Production Mode

-Dconfig.resource 的部分,會在 class path 尋找設定檔 -Dconfig.file 則是依照 file system 尋找設定檔

-J-Xms512M -J-Xmx1G -J-server 這幾個是啟動 application 的 JVM 參數

target/universal/stage/bin/test6 -Dconfig.resource=prod-application.conf -Dlogger.resource=prod-logback.xml -Dhttp.port=9000 -J-Xms512M -J-Xmx1G -J-server

target/universal/stage/bin/test6 -Dconfig.file=conf/prod-application.conf -Dlogger.file=conf/prod-logback.xml -Dhttp.port=9000 -J-Xms512M -J-Xmx1G -J-server

啟動後,會在 target/universal/stage/RUNNING_PID 產生 process ID 的檔案

如果希望在背景執行 application,就要搭配 nohup 啟動 server

nohup target/universal/stage/bin/test6 -Dconfig.resource=prod-application.conf -Dlogger.resource=prod-logback.xml -Dhttp.port=9000 -J-Xms512M -J-Xmx1G -J-server < /dev/null > /dev/null 2>&1 &

可以用 kill 的方式關掉背景執行的 server

kill -SIGTERM `cat target/universal/stage/RUNNING_PID`

kill -SIGKILL `cat target/universal/stage/RUNNING_PID`

start.sh stop.sh scripts

為了方便,我們把上面的啟動方式,做成一個 script,放在 conf 目錄中,這樣就會在 activator stage 時,同時複製到 target/universal/stage

conf/start.sh

#!/bin/bash

#export JAVA_OPTS=""
#export JAVA_OPTS="$JAVA_OPTS -Xms512M -Xmx1G -server"

console() {
    target/universal/stage/bin/test6 -Dconfig.resource=prod-application.conf -Dlogger.resource=prod-logback.xml -Dhttp.port=9000 -J-Xms512M -J-Xmx1G -J-server
}

server() {
    nohup target/universal/stage/bin/test6 -Dconfig.resource=prod-application.conf -Dlogger.resource=prod-logback.xml -Dhttp.port=9000 -J-Xms512M -J-Xmx1G -J-server < /dev/null > /dev/null 2>&1 &
}

case "$1" in

console)
    console
;;
server)
    server
;;
*)
    echo "Usage: $0 {console|server}"
;;
esac

exit 0

conf/stop.sh

#!/bin/bash

kill -SIGTERM `cat target/universal/stage/RUNNING_PID`

# kill -SIGKILL `cat target/universal/stage/RUNNING_PID`

然後就能用這樣的方式,啟動或停止 server

target/universal/stage/conf/start.sh server
target/universal/stage/conf/stop.sh

SSL

如果要讓 server 可以支援 HTTPS,我們需要做以下設定

Property Purpose Default Value
https.port https port number
https.keyStore 儲存 private key 及 certificate 的檔案路徑
https.keyStoreType key store type JavaKeyStore(JKS)
https.keyStorePassword password blank password
https.keyStoreAlgorithm key store algorithm platform's default algorithm

也可以自己實作 SSLEngine,參考 Configuring HTTPS,製作新的 class CustomSSLEngineProvider(appProvider: ApplicationProvider) extends SSLEngineProvider 。

References

Mastering Play Framework for Scala

The Application Secret

第三章 : Play建置部署與常用指令

overriding-configuration

Start and stop a Scala application in production

2016/12/19

Websocket in Scala Play

Websocket 就是在網頁 http protocol 之下,再增加的一層封裝協定,目的是讓 client 建立長時間的 socket 連線,在不重新建立連線的狀況下,我們可以傳送更多即時的資訊。

根據 WebSockets 的官方文件說明,scala play 2.5版已經調整為使用 Akka Streams 及 actors 來處理 WebSockets。

以最基本的 String 資料格式為例,跟一般的 http Action 一樣,websocket 也是以 Controller 為入口點。

WebSocketController 有兩個部分,index 是回傳一個簡單的 websocket 網頁 client,而 socket 就是主要處理 websocket 的部分。

// WebSocketController.scala
import play.api.mvc._
import play.api.libs.streams._

class WebSocketController @Inject()(implicit system: ActorSystem, materializer: Materializer) extends Controller {

  def index = Action { implicit request =>
    Ok(views.html.ws())
  }

  def socket = WebSocket.accept[String, String] { request =>
    ActorFlow.actorRef(out => MyWebSocketActor.props(out))
  }
}

ActorFlow.actorRef(...) 可以被替代為 Akka Streams Flow[In, Out, _],但還是用 actors 撰寫會比較直接。

在 WebSocketController.scala 的下面,我們直接將處理 websocket 的 MyWebSocketActor 寫在下面,收到字串就會直接回傳 echo String: 加上原本收到的字串。

import akka.actor._

object MyWebSocketActor {
  def props(out: ActorRef) = Props(new MyWebSocketActor(out))
}

class MyWebSocketActor(out: ActorRef) extends Actor {
  def receive = {
    case msg: String =>
      Logger.debug(s"echo String: ${msg}")
      out ! s"echo String: ${msg}"
  }
}

在 routes 加上兩個 service uri

GET         /ws                                         controllers.WebSocketController.socket
GET         /wstest                                     controllers.WebSocketController.index

如果不要自己撰寫 websocket client 網頁,可以直接用 Websocket Echo Test 測試,把 Location: wss://echo.websocket.org 改為 ws://localhost:9000/ws,Use secure WebSocket (TLS) 取消勾選。點 Connect 以後,就可以對 ws server 發送文字訊息。

以下是自己撰寫 websocket 網頁的 sample: ws.scala.html

<!DOCTYPE html>
<meta charset="utf-8" />
<title>WebSocket Test</title>
<script language="javascript" type="text/javascript">
    //var wsUri = "ws://localhost:9000/ws";
    //var output;

    function init()
    {
        //output = document.getElementById("output");
        //testWebSocket();
    }

    function testWebSocket()
    {
        var wsUri= document.getElementById('wsUri').value;
        websocket = new WebSocket(wsUri);
        //websocket.binaryType = 'arraybuffer';

        websocket.onopen = function(evt) { onOpen(evt) };
        websocket.onclose = function(evt) { onClose(evt) };
        websocket.onmessage = function(evt) { onMessage(evt) };
        websocket.onerror = function(evt) { onError(evt) };
    }

    function closeWebSocket() {
        websocket.close();
    }

    function onOpen(evt)
    {
        writeToScreen("CONNECTED");
        //doSend("WebSocket rocks");
    }

    function onClose(evt)
    {
        writeToScreen("DISCONNECTED");
    }

    function onMessage(evt)
    {
        console.log(evt);

        if ( evt.data instanceof ArrayBuffer ) {
            console.log( evt.data );
        } else {
            writeToScreen('<span style="color: blue;">RESPONSE: ' + evt.data+'</span>');
        }
        //websocket.close();
    }

    function onError(evt)
    {
        writeToScreen('<span style="color: red;">ERROR:</span> ' + evt.data);
    }

//    function doSend(message)
//    {
//        writeToScreen("SENT: " + message);
//        websocket.send(message);
//    }

    function doSendInput()
    {
        var message= document.getElementById('sendMessage').value;

        writeToScreen("SENT: " + message);
        websocket.send( message );
        //websocket.send( JSON.stringify(JSON.parse(message)) );
        //var byteArray = new Uint8Array(message);
        //websocket.send( byteArray.buffer );
    }

    function writeToScreen(message)
    {
        var pre = document.createElement("p");
        pre.style.wordWrap = "break-word";
        pre.innerHTML = message;

        output = document.getElementById("output");
        output.appendChild(pre);
    }

    function clearScreen()
    {
        var nodes = document.getElementById("output");
        while (nodes.hasChildNodes()) {
            nodes.removeChild(nodes.childNodes[0]);
        }
    }

    window.addEventListener("load", init, false);

    </script>

<h2>WebSocket Test</h2>

<input id="wsUri" size="35" value="ws://localhost:9000/ws"></input>

<button id="close" onclick="javascipt:testWebSocket()">connect</button>
<button id="close" onclick="javascipt:closeWebSocket()">close</button>

<br/><br/>

<input id="sendMessage" size="35" value="測試 WebSocket!!!"></input>
<button id="send" onclick="javascipt:doSendInput()">Send</button>
<button id="clear" onclick="javascipt:clearScreen()">clear</button>
<div id="output"></div>

啟動 server 瀏覽網頁 http://localhost:9000/wstest 就可以測試 websocket

如果要將資料的格式換成 json,就稍微調整一下程式。

//WebSocketController.scala 把 String 換成 JsValue

  def socketJs = WebSocket.accept[JsValue, JsValue] { request =>
    ActorFlow.actorRef(out => MyWebSocketActor.props(out))
  }

// 接收的資料改為 JsValue,並直接回傳原始的 json
class MyWebSocketActor(out: ActorRef) extends Actor {
  def receive = {
    case msg: JsValue =>
      Logger.debug(s"echo JsValue: ${msg}")
      out ! msg
  }
}

routes 修改為 socketJs method

GET         /ws                                         controllers.WebSocketController.socketJs
GET         /wstest                                     controllers.WebSocketController.index

ws.scala.html 中發送資料的部分,以 JSON.parse 及 JSON.stringify 確認發送的資料是 json。

    function doSendInput()
    {
        var message= document.getElementById('sendMessage').value;

        writeToScreen("SENT: " + message);
        //websocket.send( message );
        websocket.send( JSON.stringify(JSON.parse(message)) );
    }

測試的字串要改為 json

References

Binary websockets with Play 2.0 and Scala (and a bit op JavaCV/OpenCV)

Handling data streams reactively

Akka Streams integration in Play Framework 2.5

2016/12/12

How to use XML in scala

scala 內建了 scala.xml 可以處理 XML,以下我們簡單測試了一下 xml 的功能,目前看起來,標準函式庫的功能就足夠應用的需求了。

產生及使用 scala.xml

scala 可以直接偵測 < 這個符號,並以 xml 的方式處理,在程式中,直接把 xml指定給變數,就會直接產生 xml 物件。

scala> val xml = <store><item type="book">Novel</item><item type="pen">pencil</item></store>
xml: scala.xml.Elem = <store><item type="book">Novel</item><item type="pen">pencil</item></store>

產生 xml 物件後,可以用 text method,將所有欄位的 value 全部印出來

scala> xml.text
res0: String = Novelpencil

用 XPath 的語法,可以取得子節點的內容

scala> xml\"item"
res1: scala.xml.NodeSeq = NodeSeq(<item type="book">Novel</item>, <item type="pen">pencil</item>)

scala> (xml\"item").map(_.text).mkString(" ")
res2: String = Novel pencil

用 "@type" selector,可以取得所有 type 屬性的值

scala> (xml\"item").map(_\"@type")
res3: scala.collection.immutable.Seq[scala.xml.NodeSeq] = List(book, pen)

"@type" 只能取得目前這個節點的 children,如果要將所有出現同樣屬性的節點都取出來,就要用 "\" selector

scala> znodes\"z"
res5: scala.xml.NodeSeq = NodeSeq(<z x="1"/>)

scala> znodes\\"z"
res6: scala.xml.NodeSeq = NodeSeq(<z x="1"/>, <z x="2"/>, <z x="3"/>, <z x="4"/>)

跟剛剛一樣的方法,就可以找到屬性 x 的所有 values

scala> (znodes\\"z").map(_\"@x")
res12: scala.collection.immutable.Seq[scala.xml.NodeSeq] = List(1, 2, 3, 4)

如果是一個 xml string,可以用 scala.xml.XML.loadString 將 string 轉換為 xml。

scala> val xmlstring = """<store><item type="book">Novel</item><item type="pen">pencil</item></store>"""
xmlstring: String = <store><item type="book">Novel</item><item type="pen">pencil</item></store>

scala> val xmlfromstring = scala.xml.XML.loadString(xmlstring)
xml: scala.xml.Elem = <store><item type="book">Novel</item><item type="pen">pencil</item></store>

scala> xml == xmlfromstring
res13: Boolean = true

manual Serializing and Deserializing xml

首先定義 Stock class,在 toXml method 中,可以直接用 {symbol} 將 Stock 物件中的資料,產生到 xml 中,而 fromXml 就是用剛剛的 text 的方式取得 xml 裡面的資料放入 Stock 物件中。

case class Stock(var symbol: String, var businessName: String, var price: Double) {

  // convert Stock fields to XML
  def toXml = {
    <stock>
      <symbol>{symbol}</symbol>
      <businessName>{businessName}</businessName>
      <price>{price}</price>
    </stock>
  }

  override def toString =
    s"symbol: $symbol, businessName: $businessName, price: $price"

}

object Stock {

  // convert XML to a Stock
  def fromXml(node: scala.xml.Node):Stock = {
    val symbol = (node \ "symbol").text
    val businessName = (node \ "businessName").text
    val price = (node \ "price").text.toDouble
    new Stock(symbol, businessName, price)
  }
}

用個簡單的 TestXml 測試 toXml 及 fromXml

object TestXml extends App {

  // convert a Stock to its XML representation
  val aapl = new Stock("AAPL", "Apple", 600d)
  println(aapl.toXml)

  // convert an XML representation to a Stock
  val googXml = <stock>
    <symbol>GOOG</symbol>
    <businessName>Google</businessName>
    <price>620.00</price>
  </stock>
  val goog = Stock.fromXml(googXml)
  println(goog)
}

執行結果如下

<stock>
      <symbol>AAPL</symbol>
      <businessName>Apple</businessName>
      <price>600.0</price>
    </stock>

symbol: GOOG, businessName: Google, price: 620.0

References

Basic XML processing with Scala

Practical Scala – processing XML

Serializing and deserializing XML in Scala

Generating dynamic XML from Scala source code (like a template)

2016/12/05

Play JSON library

Scala Play 支援自己實作的 play.api.libs.json JSON Lirary

類別

  • JsValue: trait 代表任意一種 JSON value,以下這些是 extend JsValue 的 case classes

  1. JsString
  2. JsNumber
  3. JsBoolean
  4. JsObject
  5. JsArray
  6. JsNull

  • Json 提供 utilties,轉換 JsValue

  • JsPath 代表 JsValue 資料結構中的 path,類似 XML 的 XPath,可使用 pattern 在 JsValue 中進行 data traversal

Converting to a JsValue

  • 以 json String 直接 parsing 並轉換
import play.api.libs.json._

val json: JsValue = Json.parse("""
{
  "name" : "Watership Down",
  "location" : {
    "lat" : 51.235685,
    "long" : -1.309197
  },
  "residents" : [ {
    "name" : "Fiver",
    "age" : 4,
    "role" : null
  }, {
    "name" : "Bigwig",
    "age" : 6,
    "role" : "Owsla"
  } ]
}
""")

直接將上面的 code ,在 activator console 中進行測試,就可以得到測試結果

scala> json
res0: play.api.libs.json.JsValue = {"name":"Watership Down","location":{"lat":51.235685,"long":-1.309197},"residents":[{"name":"Fiver","age":4,"role":null},{"name":"Bigwig","age":6,"role":"Owsla"}]}
  • 以 class 的方式建立 JsValue
import play.api.libs.json._

val json: JsValue = JsObject(Seq(
  "name" -> JsString("Watership Down"),
  "location" -> JsObject(Seq("lat" -> JsNumber(51.235685), "long" -> JsNumber(-1.309197))),
  "residents" -> JsArray(Seq(
    JsObject(Seq(
      "name" -> JsString("Fiver"),
      "age" -> JsNumber(4),
      "role" -> JsNull
    )),
    JsObject(Seq(
      "name" -> JsString("Bigwig"),
      "age" -> JsNumber(6),
      "role" -> JsString("Owsla")
    ))
  ))
))

使用 Writes converters

scala converting to a JsValue 是以 Json.toJsonT(implicit writes: Writes[T]) 進行轉換的,這需要 Writes 物件的幫忙。Play JSON API 提供了一個 implicit Writes 物件,可以轉換 Int, Double, String, Boolean ...,同時也支援 collections

import play.api.libs.json._

// basic types
val jsonString = Json.toJson("Fiver")
val jsonNumber = Json.toJson(4)
val jsonBoolean = Json.toJson(false)

// collections of basic types
val jsonArrayOfInts = Json.toJson(Seq(1, 2, 3, 4))
val jsonArrayOfStrings = Json.toJson(List("Fiver", "Bigwig"))

如果要轉換自己的資料物件類別,就須要提供一個 implicit Writes converter

case class Location(lat: Double, long: Double)
case class Resident(name: String, age: Int, role: Option[String])
case class Place(name: String, location: Location, residents: Seq[Resident])
import play.api.libs.json._

implicit val locationWrites = new Writes[Location] {
  def writes(location: Location) = Json.obj(
    "lat" -> location.lat,
    "long" -> location.long
  )
}

implicit val residentWrites = new Writes[Resident] {
  def writes(resident: Resident) = Json.obj(
    "name" -> resident.name,
    "age" -> resident.age,
    "role" -> resident.role
  )
}

implicit val placeWrites = new Writes[Place] {
  def writes(place: Place) = Json.obj(
    "name" -> place.name,
    "location" -> place.location,
    "residents" -> place.residents)
}

val place = Place(
  "Watership Down",
  Location(51.235685, -1.309197),
  Seq(
    Resident("Fiver", 4, None),
    Resident("Bigwig", 6, Some("Owsla"))
  )
)

val json = Json.toJson(place)

也可以使用 combinator pattern

import play.api.libs.json._
import play.api.libs.functional.syntax._

implicit val locationWrites: Writes[Location] = (
  (JsPath \ "lat").write[Double] and
  (JsPath \ "long").write[Double]
)(unlift(Location.unapply))

implicit val residentWrites: Writes[Resident] = (
  (JsPath \ "name").write[String] and
  (JsPath \ "age").write[Int] and
  (JsPath \ "role").writeNullable[String]
)(unlift(Resident.unapply))

implicit val placeWrites: Writes[Place] = (
  (JsPath \ "name").write[String] and
  (JsPath \ "location").write[Location] and
  (JsPath \ "residents").write[Seq[Resident]]
)(unlift(Place.unapply))

Traversing a JsValue structure

可以 traverse JsValue 並取得特定的資料欄位,這就類似 Scala XML processing

  • Simple Path \
val lat = (json \ "location" \ "lat").get
// returns JsNumber(51.235685)
  • Recursive path \
val names = json \\ "name"
// returns Seq(JsString("Watership Down"), JsString("Fiver"), JsString("Bigwig"))
  • Index lookup (針對 JsArrays)
val bigwig = (json \ "residents")(1)
// returns {"name":"Bigwig","age":6,"role":"Owsla"}

Converting from a JsValue

  • 使用 String utilities
val minifiedString: String = Json.stringify(json)

適合閱讀的格式

val readableString: String = Json.prettyPrint(json)
  • 使用 JsValue.as/asOpt

將 JsValue 轉換為其他資料類別的方式,最簡單的方式就是用 JsValue.asT: T ,但這個方式需要用到一個 implicit converter: Reads[T]

val name = (json \ "name").as[String]
// "Watership Down"

val names = (json \\ "name").map(_.as[String])
// Seq("Watership Down", "Fiver", "Bigwig")

但是 as 可能會產生 JsResultException,所以比較好的方式,應該是改用 JsValue.asOptT: Option[T]

val nameOption = (json \ "name").asOpt[String]
// Some("Watership Down")

val bogusOption = (json \ "bogus").asOpt[String]
// None
  • 使用 validation

會產生兩種 JsResult

  1. JsSuccess: validation/convertion 成功
  2. JsError: validation/conversion 失敗
val json = { ... }

val nameResult: JsResult[String] = (json \ "name").validate[String]

// Pattern matching
nameResult match {
  case s: JsSuccess[String] => println("Name: " + s.get)
  case e: JsError => println("Errors: " + JsError.toJson(e).toString())
}

// Fallback value
val nameOrFallback = nameResult.getOrElse("Undefined")

// map
val nameUpperResult: JsResult[String] = nameResult.map(_.toUpperCase())

// fold
val nameOption: Option[String] = nameResult.fold(
  invalid = {
    fieldErrors => fieldErrors.foreach(x => {
      println("field: " + x._1 + ", errors: " + x._2)
    })
    None
  },
  valid = {
    name => Some(name)
  }
)
  • JsValue 轉換成 data model
case class Location(lat: Double, long: Double)
case class Resident(name: String, age: Int, role: Option[String])
case class Place(name: String, location: Location, residents: Seq[Resident])
import play.api.libs.json._
import play.api.libs.functional.syntax._

implicit val locationReads: Reads[Location] = (
  (JsPath \ "lat").read[Double] and
  (JsPath \ "long").read[Double]
)(Location.apply _)

implicit val residentReads: Reads[Resident] = (
  (JsPath \ "name").read[String] and
  (JsPath \ "age").read[Int] and
  (JsPath \ "role").readNullable[String]
)(Resident.apply _)

implicit val placeReads: Reads[Place] = (
  (JsPath \ "name").read[String] and
  (JsPath \ "location").read[Location] and
  (JsPath \ "residents").read[Seq[Resident]]
)(Place.apply _)


val json = { ... }

val placeResult: JsResult[Place] = json.validate[Place]
// JsSuccess(Place(...),)

val residentResult: JsResult[Resident] = (json \ "residents")(1).validate[Resident]
// JsSuccess(Resident(Bigwig,6,Some(Owsla)),)

Reference

Scala Json in Play

JSON_basics

2016/11/28

redis for scala play

scala play 預設是使用 ehcache 來當作 cache 的 library,但這個實作常常會在開發的過程當中,會因為程式自動重載的功能,而造成 cache 出現 The play Cache is not alive (STATUS_SHUTDOWN) 的錯誤訊息,在搜尋過後,了解到這可能是 ehcache 的 bug,因此就嘗試將 ehcache 更換成 redis。

安裝 redis

從開發環境到正式環境,需要在不同作業系統安裝 redis,因此我們在這裡記錄不同作業系統安裝 redis 的方法。

windows

在 windows 安裝 [料理佳餚] 在 Windows 上安裝 Redis

Redis on Windows

要注意 Memory Limit 的設定,不然 redis 應該會把記憶體吃光。

用 telnet 或是 redis-cli 可以測試有沒有裝好。

telnet localhost 6379
Mac
sudo port install redis
sudo port load redis

sudo port unload redis

# start redis manually
redis-server /opt/local/etc/redis.conf

要自己去修改 /opt/local/etc/redis.conf

# limited number of clients in developement environment
maxclients 50

# 限制 redis 最多可使用多少記憶體
maxmemory 500MB

# 當記憶體不夠時,要用什麼方式處理
maxmemory-policy volatile-lru
Debian 8

Dotdeb 是Debian 的 3rd party repository,在裡面選擇 Taiwan mirror site。

vi /etc/apt/sources.list.d/dotdeb.list

deb http://ftp.yzu.edu.tw/Linux/dotdeb/ jessie all
deb-src http://ftp.yzu.edu.tw/Linux/dotdeb/ jessie all

安裝 Dotdeb 的 GPG key

wget https://www.dotdeb.org/dotdeb.gpg
sudo apt-key add dotdeb.gpg

安裝 redis server

sudo apt-get update
sudo apt-get install redis-server

啟動/停止

sudo service redis-server start

sudo service redis-server stop

用 redis-benchmark 測試連線狀況

redis-benchmark -q -n 1000 -c 10 -P 5

調整系統參數

sudo sysctl vm.overcommit_memory=1

vi /etc/sysctl.conf
vm.overcommit_memory = 1
Centos 7

安裝 epel

wget -r --no-parent -A 'epel-release-*.rpm' http://dl.fedoraproject.org/pub/epel/7/x86_64/e/

rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-*.rpm

安裝 redis

yum install redis

啟動/停止 redis

systemctl enable redis.service
systemctl start redis.service

systemctl stop redis.service
cluster

redis cluster 是用 sharding 的方式實作的,每一個節點負責一部分 hash slot,一般要建構 cluster 環境,要用到三個 redis node,再加上備援,應該要 6 個 redis node,另外三個是 slave node。

如果要測試細節可以參考這篇文章: Redis Cluster 3.0搭建與使用

或是直接研究官方的文件 Redis cluster tutorial

redis for play

play-redis 是 Play framework 2 cache plugin as an adapter to redis-server,使用這個 framework 的優勢是實作了 play 的 CacheAPI,因此我們可以透過設定,就將 ehcache 調整為 redis。

首先要在 build.sbt 將 disable ehcache,並加上 play-redis 的 libray。

// enable Play cache API (based on your Play version) and optionally exclude EhCache implementation
libraryDependencies += play.sbt.PlayImport.cache exclude("net.sf.ehcache", "ehcache-core")
// include play-redis library
libraryDependencies += "com.github.karelcemus" %% "play-redis" % "1.3.0-M1"

接下來修改 application.conf

# disable default Play framework cache plugin
play.modules.disabled += "play.api.cache.EhCacheModule"

# enable redis cache module
play.modules.enabled += "play.api.cache.redis.RedisCacheModule"

並在 application.conf 中設定 localhost redis 的連線方式

play.cache {
  // redis for Play https://github.com/KarelCemus/play-redis
  redis {
    redis.host="localhost"
    redis.port=6379

    ##redis-server database, 1-15, default:1
    redis.database=1
    ##connection timeout, default:1s (Duration)
    redis.timeout=1s
    ## Akka actor
    redis.dispatcher="akka.actor.default-dispatcher"
    ## Defines which configuration source enable. Accepted values are static, env, custom
    redis.configuration="static"
    ## optional
    #redis.password="null"
    ## optional, Name of the environment variable with the connection string.
    #redis.connection-string-variable=""
    ## Defines behavior when command execution fails.
    #redis.recovery="log-and-default"
  }
}

透過這樣的設定,就可以在不修改 scala 程式的狀況下,就把 ehcache 改成 redis。

2016/11/21

Play cache API

Scala Play 建議使用 EHCache 作為 cache solution,這在 java 領域是一個常見的套件,目前先看看官方的 cache 方式,畢竟已經整合到 play framework 中,未來再看看有沒有辦法改用 redis。

設定 cache

build/sbt 中增加 cache 這個 libraryDependencies

libraryDependencies ++= Seq(
  cache,
  ...
)

app/controllers/CacheApplication.scala

  • cache.set 新增 cache item
  • cache.get 取得 cache item
  • cache.remove 移除 cache item
  • cache.getOrElse 取得 cache item,如果找不到就新增
package controllers

import java.util.concurrent.TimeoutException
import javax.inject.Inject

import akka.actor.ActorSystem
import akka.pattern.after
import models.{Project, ProjectRepo, TaskRepo}
import play.api.Logger
import play.api.cache.CacheApi
import play.api.libs.concurrent.Execution.Implicits.defaultContext
import play.api.mvc.{Action, Controller}

import scala.concurrent.Future
import scala.concurrent.duration._

class CacheApplication @Inject()(cache: CacheApi)
                           extends Controller {
  def newCache(name: String) = Action {
    val result = s"Add new project ${name} to cache..\n"

    val project: Project = Project(999, name)
    cache.set("project."+name, project)
    Ok(result)
  }

  def newCache2(name: String) = Action {
    val result = s"Add new project ${name} to cache with 5 minuts..\n"

    val project: Project = Project(888, name)
    cache.set("project."+name, project, 5.minutes)

    // 移除 cache item
    //cache.remove("project."+name)
    Ok(result)
  }

  def getCache(name:String) = Action {
    val project: Option[Project] = cache.get[Project]("project."+name)

    val result = project match {
      // 以 Some 測試 name 有沒有存在
      case Some(pj) => s"get project from cache: ${pj.name}"
      case None => s"can't get project from cache.."
    }

    Ok(result)
  }

  def getCache2(name:String) = Action {
    val project: Project = cache.getOrElse[Project]("project."+name) {
      // 如果 cache 中找不到 project.name 就產生一個新的,存到 cache 中
      Project(777, name)
    }

    val result = s"getOrElse project from cache: ${project.name}"
    Ok(result)
  }

}

conf/routes

GET           /cache/:name               controllers.CacheApplication.newCache(name:String)
GET           /cache2/:name               controllers.CacheApplication.newCache2(name:String)
GET           /getcache/:name            controllers.CacheApplication.getCache(name:String)
GET           /getcache2/:name           controllers.CacheApplication.getCache2(name:String)

測試

$ curl 'http://localhost:9000/cache/cproject'
Add new project cproject to cache..

$ curl 'http://localhost:9000/getcache/cproject'
get project from cache: cproject

$ curl 'http://localhost:9000/cache2/cproject'
Add new project cproject to cache with 5 minuts..

$ curl 'http://localhost:9000/getcache2/cproject'
getOrElse project from cache: cproject

預設的 cache store

預設緩存叫 play, 並可以利用 ehcache.xml 來設定新的 cache store。

調整 application.conf

play.cache {
  # If you want to bind several caches, you can bind the individually
  bindCaches = ["db-cache", "user-cache", "session-cache"]
}

在 controller/Application.scala 中使用新的 cache store

import play.api.cache._
import play.api.mvc._
import javax.inject.Inject

class Application @Inject()(
    @NamedCache("session-cache") sessionCache: CacheApi
) extends Controller {

}

Caching HTTP responses

可以將 HTTP response 放進 cache

首先利用 cached: Cached 來 cache actions

import play.api.cache.Cached
import javax.inject.Inject

class Application @Inject() (cached: Cached) extends Controller {

}

例如這個方式,就是將 home 放入 cache

  def cacheAction = cached("homePage") {
    Action {
      Ok("Hello World")
    }
  }

也可以搭配 Authenticated,為每一個 user 暫存不同的 result

def userProfile = Authenticated {
  user =>
    cached(req => "profile." + user) {
      Action {
        Ok(views.html.profile(User.find(user)))
      }
    }
}

可以選擇只要 cache 200 OK 的 response

def get(index: Int) = cached.status(_ => "/resource/"+ index, 200) {
  Action {
    if (index > 0) {
      Ok(Json.obj("id" -> index))
    } else {
      NotFound
    }
  }
}

或是將 404 Not Found 暫存幾分鐘

def get(index: Int) = {
  val caching = cached
    .status(_ => "/resource/"+ index, 200)
    .includeStatus(404, 600)

  caching {
    Action {
      if (index % 2 == 1) {
        Ok(Json.obj("id" -> index))
      } else {
        NotFound
      }
    }
  }
}

Custom Cache API

如果要自己實作新的 Cache API

要先在 application.conf disable EhCacheModule

play.modules.disabled += "play.api.cache.EhCacheModule"

然後用新的 cache API implementation,並 reuse NamedCache 綁定該 implementation。

References

Scala Cache

Play 緩存 API

2016/11/14

slick Database Persistence in Scala play 2.5

在 scala play 2.5 framework 中,要將資料儲存在 DB 中有好幾種方式,都是以整合第三方套件的方式實作,我們測試了原始的 JDBC 以及 slick 兩種,以下是 slick 的部分。

slick 是 functional relational mapping database library,是以 functional programming 方式存取關聯式資料庫。

準備 slick project

首先以 activator 產生一個新的 project,我們是使用Play Framework 2.5 and Slick 3.1

activator new tst6 play-slick3-example

這個 template 原本是使用 H2 memory database,改用 mysql。

build.sbt

name := """play-slick-example"""

version := "1.0"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.11.7"

routesGenerator := InjectedRoutesGenerator

resolvers += "scalaz-bintray" at "https://dl.bintray.com/scalaz/releases"

libraryDependencies ++= Seq(
    cache,
    ws,
    filters,
    "com.typesafe.play" %% "play-slick" % "2.0.0",
    "com.typesafe.play" %% "play-slick-evolutions" % "2.0.0",
    //"com.h2database" % "h2" % "1.4.187",
    "mysql" % "mysql-connector-java" % "5.1.36",
    "org.scalatestplus.play" %% "scalatestplus-play" % "1.5.0" % "test",
    specs2 % Test
)

resolvers += "Sonatype snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/"

fork in run := true

修改 logback.xml,主要是增加以下這一行的設定。

<logger name="slick.jdbc.JdbcBackend.statement"  level="DEBUG" />

完整的 loback.xml 內容如下

<!-- https://www.playframework.com/documentation/latest/SettingsLogger -->
<configuration>

<conversionRule conversionWord="coloredLevel" converterClass="play.api.libs.logback.ColoredLevel" />

<!--
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>${application.home:-.}/logs/application.log</file>
    <encoder>
        <pattern>%date [%level] from %logger in %thread\n\t%message%n%xException</pattern>
    </encoder>
</appender>
-->

<appender name="FILE"
          class="ch.qos.logback.core.rolling.RollingFileAppender">
    <append>true</append>
    <rollingPolicy
            class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <param name="FileNamePattern"
               value="${application.home:-.}/logs/application.%d{yyyy-MM-dd}.log.zip">
        </param>
    </rollingPolicy>
    <encoder>
        <!-- <pattern>%d %-5p %c %L%n %m%n</pattern> -->
        <!-- <charset class="java.nio.charset.Charset">UTF-8</charset>  -->
        <pattern>%date [%level] from %logger in %thread\n\t%message%n%xException</pattern>
    </encoder>
</appender>

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%coloredLevel %logger{15} in %thread\n\t%message%n%xException{10}</pattern>
    </encoder>
</appender>

<appender name="ASYNCFILE" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="FILE" />
</appender>

<appender name="ASYNCSTDOUT" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="STDOUT" />
</appender>

<logger name="play" level="INFO" />
<logger name="application" level="DEBUG" />

<!-- Will log all statements -->
<logger name="slick.jdbc.JdbcBackend.statement"  level="DEBUG" />

<!-- Off these ones as they are annoying, and anyway we manage configuration ourselves -->
<logger name="com.avaje.ebean.config.PropertyMapLoader" level="OFF" />
<logger name="com.avaje.ebeaninternal.server.core.XmlConfigLoader" level="OFF" />
<logger name="com.avaje.ebeaninternal.server.lib.BackgroundThread" level="OFF" />
<logger name="com.gargoylesoftware.htmlunit.javascript" level="OFF" />

<root level="INFO">
    <appender-ref ref="ASYNCFILE" />
    <appender-ref ref="ASYNCSTDOUT" />
</root>

</configuration>

project 設定

修改 application.conf,slick 並不是使用 JDBC 的 DB connection,要另外設定 slick.dbs

#slick.dbs.default.driver="slick.driver.H2Driver$"
#slick.dbs.default.db.driver=org.h2.Driver
#slick.dbs.default.db.url="jdbc:h2:mem:play;DB_CLOSE_DELAY=-1"
//slick.dbs.default.db.user=user
//slick.dbs.default.db.password=""

slick.dbs.default.driver="slick.driver.MySQLDriver$"
slick.dbs.default.db.driver=com.mysql.jdbc.Driver
slick.dbs.default.db.url="jdbc:mysql://localhost:3306/playdb?useUnicode=true&amp;characterEncoding=utf-8"
slick.dbs.default.db.user="root"
slick.dbs.default.db.password="max168kit"
# HikariCP connection pool the min size is numThreads, and the max size is numThreads * 5
slick.dbs.default.db.numThreads=5
slick.dbs.default.db.queueSize=30
slick.dbs.default.db.connectionTimeout=15s
slick.dbs.default.db.connectionTestQuery="select 1"

ref: connection pool

DB evolution

slick 的 db evolution 跟 JDBC 的部分一樣。

application.conf

play.evolutions {
  # You can disable evolutions for a specific datasource if necessary
  db.default.enabled = true
  autoApply = true
  autoApplyDowns = true
}

conf/evolutons.default/1.sql

# DC schema
 
# --- !Ups


CREATE TABLE PROJECT (
    ID integer NOT NULL AUTO_INCREMENT PRIMARY KEY,
    NAME varchar(255) NOT NULL
);


CREATE TABLE TASK (
    ID integer NOT NULL AUTO_INCREMENT PRIMARY KEY,
    COLOR varchar(255) NOT NULL,
    STATUS varchar(255) NOT NULL,
    PROJECT integer NOT NULL,
    FOREIGN KEY (PROJECT) REFERENCES PROJECT (ID)
);


 
# --- !Downs

DROP TABLE TASK;
DROP TABLE PROJECT;

scala codes

這個部分的 code 都是由 play-slick3-example 這個 template 來的,並沒有做什麼修改,主要可以發現,Application 都是使用 Action.async 搭配 Future 的方式,進行非同步的處理。

conf/routes URI 的設定用到了 PUT, PATCH 比較少見的 HTTP Method,我們在測試時,可以用 Chrome Postman 進行測試。

GET           /                          controllers.Application.listProjects
PUT           /projects/:name            controllers.Application.createProject(name: String)
GET           /projects/list             controllers.Application.listProjects
GET           /projects/:id              controllers.Application.projects(id: Long)
PUT           /projects/:id/:name        controllers.Application.addTaskToProject(name: String, id: Long)
PATCH         /tasks/:id                 controllers.Application.modifyTask(id: Long, color:Option[String] ?= None)

DELETE        /projects/:name            controllers.Application.delete(name: String)
  • app/controllers/Application.scala
package controllers

import java.util.concurrent.{TimeoutException, TimeUnit}
import javax.inject.Inject

import akka.actor.ActorSystem
import models.{Project, ProjectRepo, TaskRepo}
import play.api.Logger
import play.api.libs.concurrent.Execution.Implicits.defaultContext
import play.api.mvc.{Action, Controller}

import akka.pattern.after
import scala.concurrent.duration._
import scala.concurrent.Future

class Application @Inject()( projectRepo: ProjectRepo, taskRepo: TaskRepo, actorSystem: ActorSystem)
                           extends Controller {

  def addTaskToProject(color: String, projectId: Long) = Action.async { implicit rs =>
    projectRepo.addTask(color, projectId)
      .map{ _ =>  Redirect(routes.Application.projects(projectId)) }
  }

  def modifyTask(taskId: Long, color: Option[String]) = Action.async { implicit rs =>
    taskRepo.partialUpdate(taskId, color, None, None).map(i =>
    Ok(s"Rows affected : $i"))
  }
  def createProject(name: String)= Action.async { implicit rs =>
    projectRepo.create(name)
      .map(id => Ok(s"project $id created") )
  }

  def listProjects = Action.async { implicit rs =>
    projectRepo.all
      .map(projects => Ok(views.html.projects(projects)))
  }

  def projects(id: Long) = Action.async { implicit rs =>
    for {
      Some(project) <-  projectRepo.findById(id)
      tasks <- taskRepo.findByProjectId(id)
    } yield Ok(views.html.project(project, tasks))
  }

  def delete(name: String) = Action.async { implicit rs =>
    projectRepo.delete(name).map(num => Ok(s"$num projects deleted"))
  }
}
  • app/models/Project.scala

ProjectsTable 的部分是在定義 Projects

package models

import javax.inject.Inject
import play.api.Logger
import play.api.db.slick.DatabaseConfigProvider
import slick.dbio
import slick.dbio.Effect.Read
import slick.driver.JdbcProfile
import slick.jdbc.GetResult
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future

case class Project(id: Long, name: String)


class ProjectRepo @Inject()(taskRepo: TaskRepo)(protected val dbConfigProvider: DatabaseConfigProvider) {

  val dbConfig = dbConfigProvider.get[JdbcProfile]
  val db = dbConfig.db
  import dbConfig.driver.api._
  private val Projects = TableQuery[ProjectsTable]

  private def _findById(id: Long): DBIO[Option[Project]] =
    Projects.filter(_.id === id).result.headOption

  private def _findByName(name: String): Query[ProjectsTable, Project, List] =
    Projects.filter(_.name === name).to[List]

  def findById(id: Long): Future[Option[Project]] =
    db.run(_findById(id))

  def findByName(name: String): Future[List[Project]] =
    db.run(_findByName(name).result)

  def all: Future[List[Project]] =
    db.run(Projects.to[List].result)

  def create(name: String): Future[Long] = {
    val project = Project(0, name)
    db.run(Projects returning Projects.map(_.id) += project)
  }

  def delete(name: String): Future[Int] = {
    val query = _findByName(name)

    val interaction = for {
      projects        <- query.result
      _               <- DBIO.sequence(projects.map(p => taskRepo._deleteAllInProject(p.id)))
      projectsDeleted <- query.delete
    } yield projectsDeleted

    db.run(interaction.transactionally)
  }

  def addTask(color: String, projectId: Long): Future[Long] = {
    val interaction = for {
      Some(project) <- _findById(projectId)
      id <- taskRepo.insert(Task(0, color, TaskStatus.ready, project.id))
    } yield id

    db.run(interaction.transactionally)
  }


  // 定義 Project 這個 table
  private class ProjectsTable(tag: Tag) extends Table[Project](tag, "PROJECT") {

    // primary key 為 ID
    def id = column[Long]("ID", O.AutoInc, O.PrimaryKey)
    def name = column[String]("NAME")

    def * = (id, name) <> (Project.tupled, Project.unapply)
    def ? = (id.?, name.?).shaped.<>({ r => import r._; _1.map(_ => Project.tupled((_1.get, _2.get))) }, (_: Any) => throw new Exception("Inserting into ? projection not supported."))

  }
}
  • app/models/Task.scala

比較特別的地方,是 TaskStatus,還有 taskStatusColumnType 做自動轉換,這裡對應到一個 DB 欄位,有 ready/set/go 這三種數值,所以用 Enumeration 定義該欄位。

package models

import javax.inject.Inject

import play.api.db.slick.DatabaseConfigProvider

import slick.driver.JdbcProfile

import scala.concurrent.Future

case class Task(id: Long, color: String, status: TaskStatus.Value, project: Long) {

  def patch(color: Option[String], status: Option[TaskStatus.Value], project: Option[Long]): Task =
    this.copy(color = color.getOrElse(this.color),
              status = status.getOrElse(this.status),
              project = project.getOrElse(this.project))

}

object TaskStatus extends Enumeration {
  val ready = Value("ready")
  val set = Value("set")
  val go = Value("go")
}

class TaskRepo @Inject()(protected val dbConfigProvider: DatabaseConfigProvider) {
  val dbConfig = dbConfigProvider.get[JdbcProfile]
  val db = dbConfig.db
  import dbConfig.driver.api._
  private val Tasks = TableQuery[TasksTable]


  def findById(id: Long): Future[Task] =
    db.run(Tasks.filter(_.id === id).result.head)

  def findByColor(color: String): DBIO[Option[Task]] =
    Tasks.filter(_.color === color).result.headOption

  def findByProjectId(projectId: Long): Future[List[Task]] =
    db.run(Tasks.filter(_.project === projectId).to[List].result)

  def findByReadyStatus: DBIO[List[Task]] =
    Tasks.filter(_.status === TaskStatus.ready).to[List].result

  def partialUpdate(id: Long, color: Option[String], status: Option[TaskStatus.Value], project: Option[Long]): Future[Int] = {
    import scala.concurrent.ExecutionContext.Implicits.global

    val query = Tasks.filter(_.id === id)

    val update = query.result.head.flatMap {task =>
      query.update(task.patch(color, status, project))
    }

    db.run(update)
  }

  def all(): DBIO[Seq[Task]] =
    Tasks.result

  def insert(Task: Task): DBIO[Long] =
    Tasks returning Tasks.map(_.id) += Task

  def _deleteAllInProject(projectId: Long): DBIO[Int] =
    Tasks.filter(_.project === projectId).delete

  private class TasksTable(tag: Tag) extends Table[Task](tag, "TASK") {

    def id = column[Long]("ID", O.AutoInc, O.PrimaryKey)
    def color = column[String]("COLOR")
    def status = column[TaskStatus.Value]("STATUS")
    def project = column[Long]("PROJECT")

    def * = (id, color, status, project) <> (Task.tupled, Task.unapply)
    def ? = (id.?, color.?, status.?, project.?).shaped.<>({ r => import r._; _1.map(_ => Task.tupled((_1.get, _2.get, _3.get, _4.get))) }, (_: Any) => throw new Exception("Inserting into ? Taskion not supported."))
  }

  implicit val taskStatusColumnType = MappedColumnType.base[TaskStatus.Value, String](
    _.toString, string => TaskStatus.withName(string))

}

測試

Postman 是一個 Chome APP,可以進行 http 測試

plain SQL in slick

如果要在 slick 裡面使用 SQL,則用別的方式進行。

ref: slick plain sql play-slick 版本對應

activator 中有個 sample template,但只拿來參考而已。

activator new test7 slick-plainsql-3.0
  • conf/routes

首先在 routes 的地方加上 URI

GET           /pj/:id                    controllers.Application.getproject(id:Long)
GET           /pj2/:id                   controllers.Application.getproject2(id:Long)
GET           /pj3/:id                   controllers.Application.getproject3(id:Long)

GET           /update/:id/:name          controllers.Application.updateproject(id:Long, name:String)

Application 的 constructor 要 @Inject() actorSystem: ActorSystem,getproject 的部分是在測試 Future 的寫法,這個部分故意留下錯誤的寫法,因為 Future 區塊裡面的 callback codes,是使用不同的 thread 執行的。

getproject3 加上了非同步的 timeout 檢查,如果 2 seconds 後沒有完成,就會產生 Excetpion。

package controllers

import java.util.concurrent.{TimeoutException, TimeUnit}
import javax.inject.Inject

import akka.actor.ActorSystem
import models.{Project, ProjectRepo, TaskRepo}
import play.api.Logger
import play.api.libs.concurrent.Execution.Implicits.defaultContext
import play.api.mvc.{Action, Controller}

import akka.pattern.after
import scala.concurrent.duration._
import scala.concurrent.Future

class Application @Inject()( projectRepo: ProjectRepo, taskRepo: TaskRepo, actorSystem: ActorSystem)
                           extends Controller {

  def getproject(id:Long) = Action {
    Logger.info(s"getproject id=${id}")
    var result = "DB project:\n"

    val pjs: Future[Seq[String]] = projectRepo.findByIdCustom2(id)
    pjs.map{
          // 因為是非同步,這裡是在不同 thread 執行的
      cs => {
        for(c<-cs) {
          Logger.info("c="+c.toString)
          // c=test
          result += c.toString
          Logger.info(s"result=$result")
        }
      }
    }
    // 這是錯誤的寫法
    // 只有 DB project:  沒有 db 查詢的結果....  錯誤 的結果
    Logger.info(s"result=$result")
    Ok(result)
  }

  def getproject2(id: Long) = Action.async {

    //val futureNumRowsDeleted = scala.concurrent.Future{ Transaction.delete(id) }
    val pjs: Future[Seq[String]] = projectRepo.findByIdCustom2(id)

    pjs.map {
      var result = "DB project:\n"
      cs => {
        for (c <- cs) {
          Logger.info("c=" + c.toString)
          // c=test
          result += c.toString+" "
          Logger.info(s"result=$result")
        }
      }
        Logger.info(s"result=$result")
        Ok(result)
    }
  }

  def getproject3(id: Long) = Action.async {

    //val futureNumRowsDeleted = scala.concurrent.Future{ Transaction.delete(id) }
    val pjs: Future[Seq[Project]] = projectRepo.findByIdCustom3(id)

    //val timeout = play.api.libs.concurrent.Promise.timeout("Past max time", 2, TimeUnit.SECONDS)

    //val timeoutFuture = after(2.second, actorSystem.scheduler)(Future.successful("Oops"))
    val timeoutFuture = after(2.second, actorSystem.scheduler)(Future.failed(new TimeoutException("Future timed out!")))

    Future.firstCompletedOf(Seq(pjs, timeoutFuture)).map {
      case cs: Seq[Project]  => {
        var result = "DB project:\n"
        for (c <- cs) {
          Logger.info("c=" + c.name)
          // c=test
          result += c.name + " "
          Logger.info(s"result=$result")
        }
        Ok(result)
      }
      case t: TimeoutException => InternalServerError(t.getMessage)
    }
  }
  
  def updateproject(id: Long, name:String) = Action.async {
    val pjs: Future[Int] = projectRepo.updateproject(id, name)

    val timeoutFuture = after(2.second, actorSystem.scheduler)(Future.failed(new TimeoutException("Future timed out!")))

    Future.firstCompletedOf(Seq(pjs, timeoutFuture)).map {
      case cs: Int  => {
        val result = s"DB update result:${cs}\n"
        Ok(result)
      }
      //case t: Any => InternalServerError()
    }
  }
}

app/models/Project.scala

findByIdCustom3 是在測試直接轉換成 Project 物件的方法。

package models

import javax.inject.Inject
import play.api.Logger
import play.api.db.slick.DatabaseConfigProvider
import slick.dbio
import slick.dbio.Effect.Read
import slick.driver.JdbcProfile
import slick.jdbc.GetResult
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future

case class Project(id: Long, name: String)


class ProjectRepo @Inject()(taskRepo: TaskRepo)(protected val dbConfigProvider: DatabaseConfigProvider) {

  val dbConfig = dbConfigProvider.get[JdbcProfile]
  val db = dbConfig.db
  import dbConfig.driver.api._
  private val Projects = TableQuery[ProjectsTable]

  /////////////

  def findByIdCustom2(id:Long): Future[Seq[String]] = {
    val query = sql"select NAME from PROJECT where ID=$id".as[(String)]
    //db.run(query)
    Logger.info("findByIdCustom2")
    //db.run(query)

    val f: Future[Seq[String]] = db.run(query)

    //f.onSuccess { case s => println(s"Result: $s") }

    f
  }

  implicit val getProjectResult = GetResult(r => Project(r.nextLong, r.nextString))

  def findByIdCustom3(id:Long): Future[Seq[Project]] = {
    // as[(Project)] 的部分會參考到 上面的 getProjectResult 的 GetResult,並自動轉換為 Project 物件
    val query = sql"select ID, NAME from PROJECT where ID=$id".as[(Project)]
    //db.run(query)
    Logger.info("findByIdCustom3")
    //db.run(query)

    val f: Future[Seq[Project]] = db.run(query)

    //f.onSuccess { case s => println(s"Result: $s") }

    f
  }
  
  def updateproject(id:Long, name:String): Future[Int] = {
    val update = sqlu"update PROJECT set name=$name where ID=$id"
    //db.run(query)
    Logger.info("updateproject")
    //db.run(query)

    val f: Future[Int] = db.run(update)

    //f.onSuccess { case s => println(s"Result: $s") }

    f
  }
}

測試就直接用 curl 就可以了

curl -v 'http://localhost:9000/pj/1'
curl -v 'http://localhost:9000/pj2/1'
curl -v 'http://localhost:9000/pj3/1'

curl -v 'http://localhost:9000/update/1/test2'

2016/11/07

JDBC Database Persistence in Scala play 2.5

在 scala play 2.5 framework 中,要將資料儲存在 DB 中有好幾種方式,都是以整合第三方套件的方式實作,我們測試了原始的 JDBC 以及 slick 兩種,以下是 JDBC 的部分。

準備資料庫

雖然很多範例都是以 H2 memory database 展示,不過我們還是以 Maria DB 進行測試,首先要準備資料庫。

  • create mysql database: playdb
CREATE DATABASE playdb DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
  • create table: user
CREATE TABLE `user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`login_id` varchar(45) NOT NULL,
`password` varchar(50) NOT NULL,
`name` varchar(45) DEFAULT NULL,
`dob` bigint(20) DEFAULT NULL,
`is_active` tinyint(1) NOT NULL DEFAULT '1',
PRIMARY KEY (`id`),
UNIQUE KEY `login_id_UNIQUE` (`login_id`),
UNIQUE KEY `id_UNIQUE` (`id`)
)

BEGIN;
INSERT INTO `user` (login_id, password, name) VALUES ('test1', 'test1', 'test1'), ('test2', 'test2', 'test2');
COMMIT;

準備 scala play project

  • 以 activator 產生一個 scala play project
activator new test5 play-scala
  • 修改 build.sbt

增加 jdbc 及 "mysql" % "mysql-connector-java" % "5.1.36",

name := """test5"""

version := "1.0-SNAPSHOT"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  jdbc,
  cache,
  ws,
  filters,
  "mysql" % "mysql-connector-java" % "5.1.36",
  "org.scalatestplus.play" %% "scalatestplus-play" % "1.5.1" % Test
)

resolvers += "scalaz-bintray" at "http://dl.bintray.com/scalaz/releases"
  • application.conf 增加 default db 的設定

scala play 是以 HikariCP 作為 DB connection pool library,可以在 application.conf 中調整 DB 及 connection pool 設定。

play.db {
  config = "db"
  default = "default"

  prototype {
    # Sets a fixed JDBC connection pool size of 2
    hikaricp.minimumIdle = 2
    hikaricp.maximumPoolSize = 5
  }
}

db {

  default.driver=com.mysql.jdbc.Driver
  default.url="jdbc:mysql://localhost:3306/playdb?useUnicode=true&amp;characterEncoding=utf-8"
  default.username = "root"
  default.password = "password"

  default.logSql=true
}

要確認 connection pool 有沒有作用,可以直接在 MySQL client 中,用以下的指令查看 DB 的資訊。

# 查看資料庫狀態
show status;

# 列出連線的數量

show status where `variable_name` = 'Threads_connected';

# 列出有哪些session
show processlist;

# 列出所有的session
show full processlist;
# 列出目前連線最多的 IP
mysql -uroot -p -e "show processlist"|awk '{print $3}' |awk -F: '{print $1}' |sort |uniq -c |sort -nr
  • 修改 routes

因為是測試而已,簡單將 Application 的 Action 各自對應到獨立的 URI。

GET        /                     controllers.Application.index

GET        /dbUser               controllers.Application.fetchDBUser

GET        /dbUser2              controllers.Application.fetchDBUser2

GET        /addUser/:name        controllers.Application.addUser(name:String)
  • 修改 controllers/Application.scala

注意 Application 的 constructor,因為 Play 2.5 已經使用 Inject 的方式取得資源,我們要使用的 DB 要在 constructor 中引用進來: @NamedDatabase("default") db: Database。

後面使用 db 的部分,就跟一般的 JDBC 差不多,createStament 以後,再執行 query 或是 update。

package controllers

import javax.inject.Inject

import play.api.mvc._
import play.api.db._

class Application @Inject()(@NamedDatabase("default") db: Database) extends Controller {

  def index = Action {
    Ok(views.html.main())
  }

  def fetchDBUser = Action {
    var result = "DB User:\n"
    val conn = db.getConnection()
    try {
      val rs = conn.createStatement().executeQuery("SELECT * from user")
      while (rs.next()) {
        result += rs.getString("login_id") + "\n"
      }
    } finally {
      conn.close()
    }
    Ok(result)
  }

  // without try blocks,db.withConnection 的 connection 會在結束時自動關閉
  def fetchDBUser2 = Action {
    var result = "DB User:" + "\n"
    db.withConnection { conn =>
      val rs = conn.createStatement().executeQuery("SELECT * from user")
      while (rs.next()) {
        result += rs.getString("login_id") + "\n"
      }
    }
    Ok(result)
  }

  def addUser(name: String) = Action {
    db.withTransaction { conn =>
      val rs = conn.createStatement().executeUpdate(s"insert into user (login_id, password, name) values('$name', '$name', '$name')")
    }
    Ok
  }

}

測試

> curl 'http://localhost:9000/dbUser'
DB User:
test1
test2
> curl 'http://localhost:9000/dbUser2'
DB User:
test1
test2
> curl -v 'http://localhost:9000/addUser/test3'
*   Trying ::1...
* Connected to localhost (::1) port 9000 (#0)
> GET /addUser/test4 HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.50.1
> Accept: */*
>
< HTTP/1.1 200 OK
< X-ExampleFilter: foo
< Content-Length: 0
< Date: Thu, 11 Aug 2016 07:58:02 GMT
<
* Connection #0 to host localhost left intact

JDBC 的 logback 設定

如果要讓 log 記錄 SQL statement,就修改 logback.xml,增加這三行,但不建議在 production 環境加上這個設定

  <logger name="org.jdbcdslog.ConnectionLogger" level="OFF"  /> <!-- Won' log connections -->
  <logger name="org.jdbcdslog.StatementLogger"  level="INFO" /> <!-- Will log all statements -->
  <logger name="org.jdbcdslog.ResultSetLogger"  level="OFF"  /> <!-- Won' log result sets -->

完整的 logback.xml 內容如下

<!-- https://www.playframework.com/documentation/latest/SettingsLogger -->
<configuration>

  <conversionRule conversionWord="coloredLevel" converterClass="play.api.libs.logback.ColoredLevel" />

<!--
  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>${application.home:-.}/logs/application.log</file>
    <encoder>
      <pattern>%date [%level] from %logger in %thread\n\t%message%n%xException</pattern>
    </encoder>
  </appender>
-->

<appender name="FILE"
          class="ch.qos.logback.core.rolling.RollingFileAppender">
    <append>true</append>
    <rollingPolicy
            class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <param name="FileNamePattern"
               value="${application.home:-.}/logs/application.%d{yyyy-MM-dd}.log.zip">
        </param>
    </rollingPolicy>
    <encoder>
        <!-- <pattern>%d %-5p %c %L%n %m%n</pattern> -->
        <!-- <charset class="java.nio.charset.Charset">UTF-8</charset>  -->
        <pattern>%date [%level] from %logger in %thread\n\t%message%n%xException</pattern>
    </encoder>
</appender>

  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>%coloredLevel %logger{15} in %thread\n\t%message%n%xException{10}</pattern>
    </encoder>
  </appender>

  <appender name="ASYNCFILE" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="FILE" />
  </appender>

  <appender name="ASYNCSTDOUT" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="STDOUT" />
  </appender>

  <logger name="play" level="INFO" />
  <logger name="application" level="DEBUG" />

  <logger name="org.jdbcdslog.ConnectionLogger" level="OFF"  /> <!-- Won' log connections -->
  <logger name="org.jdbcdslog.StatementLogger"  level="INFO" /> <!-- Will log all statements -->
  <logger name="org.jdbcdslog.ResultSetLogger"  level="OFF"  /> <!-- Won' log result sets -->

  <!-- Off these ones as they are annoying, and anyway we manage configuration ourselves -->
  <logger name="com.avaje.ebean.config.PropertyMapLoader" level="OFF" />
  <logger name="com.avaje.ebeaninternal.server.core.XmlConfigLoader" level="OFF" />
  <logger name="com.avaje.ebeaninternal.server.lib.BackgroundThread" level="OFF" />
  <logger name="com.gargoylesoftware.htmlunit.javascript" level="OFF" />

  <root level="WARN">
    <appender-ref ref="ASYNCFILE" />
    <appender-ref ref="ASYNCSTDOUT" />
  </root>

</configuration>

這樣就可以在 log 中看到 SQL statement,當然這個設定只能用在 Dev 環境,不適合用在 Production。

[info] o.j.StatementLogger in application-akka.actor.default-dispatcher-2
    java.sql.Statement.executeQuery: SELECT * from user;
[info] o.j.StatementLogger in application-akka.actor.default-dispatcher-3
    java.sql.Statement.executeUpdate: insert into user (login_id, password, name) values('test4', 'test4', 'test4');

Database Evolution

scala play 內建了 db 升降版本的功能,如果要使用,必須先準備一個全新的空白的資料庫,但先不要加上 tables。

  • build.sbt 中增加 libraryDependencies += evolutions
name := """test5"""

version := "1.0-SNAPSHOT"

lazy val root = (project in file(".")).enablePlugins(PlayScala)

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  jdbc,
  evolutions,
  cache,
  ws,
  filters,
  "mysql" % "mysql-connector-java" % "5.1.36",
  "com.typesafe.play" %% "anorm" % "2.5.0",
  //"com.typesafe.play" %% "play-slick" % "2.0.0",
  //"com.typesafe.play" %% "play-slick-evolutions" % "2.0.0",
  "org.scalatestplus.play" %% "scalatestplus-play" % "1.5.1" % Test
)

resolvers += "scalaz-bintray" at "http://dl.bintray.com/scalaz/releases"

application.conf 增加 ply.evolutions 這個部分的設定

play.evolutions {
  # You can disable evolutions for a specific datasource if necessary
  db.default.enabled = true
  autoApply = true
  autoApplyDowns = true
}
  • 新增兩個 sql file

DB evolution 是以 sql file 的方式,進行 DB 版本升降,要注意 sql file 裡面規定一定要有 Ups 以及 Downs 這兩個部分。

/conf/evolutions/default/1.sql

# Users schema

# --- !Ups

CREATE TABLE `user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`login_id` varchar(45) NOT NULL,
`password` varchar(50) NOT NULL,
`name` varchar(45) DEFAULT NULL,
`dob` bigint(20) DEFAULT NULL,
`is_active` tinyint(1) NOT NULL DEFAULT '1',
PRIMARY KEY (`id`),
UNIQUE KEY `login_id_UNIQUE` (`login_id`),
UNIQUE KEY `id_UNIQUE` (`id`)
);

INSERT INTO `user` (login_id, password, name) VALUES ('test1', 'test1', 'test1'), ('test2', 'test2', 'test2');


# --- !Downs

DROP TABLE user;

/conf/evolutions/default/2.sql

# Users schema

# --- !Ups

CREATE TABLE temp (
    id bigint(20) NOT NULL AUTO_INCREMENT,
    email varchar(255) NOT NULL,
    password varchar(255) NOT NULL,
    fullname varchar(255) NOT NULL,
    isAdmin boolean NOT NULL,
    PRIMARY KEY (id)
);

# --- !Downs

DROP TABLE temp;
  • 測試 evolution

啟動 play,如果有了第一個 DB 連線,就會進行 1.sql, 2.sql 建立兩個 tables,如果想要降版,就把 2.sql 改成 2.sql.bak,server code reload 後,有了第一個 DB 連線,就會自動降版。

mysql DB 裡面會自動產生一個 table: play_evolutions,他會記錄每一次 DB 升降版本執行的 db script。

DROP TABLE IF EXISTS `play_evolutions`;
CREATE TABLE `play_evolutions` (
  `id` int(11) NOT NULL,
  `hash` varchar(255) NOT NULL,
  `applied_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `apply_script` mediumtext,
  `revert_script` mediumtext,
  `state` varchar(255) DEFAULT NULL,
  `last_problem` mediumtext,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Play DB Evolutions

2016/10/24

HTTP programming in Scala Play

如何在 scala play 中取得 broswer 的 request,並將結果回傳給 browser,scala play 是以 routes 設定為進入點,確認要使用哪一個 controller 處理哪一個 url request,最重要的是,簡化了非同步處理的處理方法,我們可以很輕鬆地就隔絕後端 DB 的處理,避免因 DB 異常影響到前端頁面卡頓的狀況。

Actions, Controllers and Results

在 Play 收到的 browser request 就是由 Action 處理,並產生 Result

plya.api.mvc.Action 就等同於 (play.api.mvc.Request => plya.api.mv.Result) 函數

controller 就是 Action generator,在 controller 中可以定義 method 為

package controllers

import play.api.mvc._

class Application extends Controller {
  def echo = Action { request =>
    Ok("Got request [" + request + "]")
  }
  
  def hello(name: String) = Action {
    Ok("Hello " + name)
  }
}

產生 Action 的範例

Action {
  Ok("Hello world")
}

Action { request =>
  Ok("Got request [" + request + "]")
}

Action { implicit request =>
  Ok("Got request [" + request + "]")
}

// 指定 BodyParser
Action(parse.json) { implicit request =>
  Ok("Got request [" + request + "]")
}

index 及 index2 會產生相同的頁面 Result

import play.api.http.HttpEntity

def index = Action {
  Result(
    header = ResponseHeader(200, Map.empty),
    body = HttpEntity.Strict(ByteString("Hello world!"), Some("text/plain"))
  )
}

def index2 = Action {
  Ok("Hello world!")
}

以下這些為各種不同類型的 play.api.mvc.Results 的範例

val ok = Ok("Hello world!")
val notFound = NotFound
val pageNotFound = NotFound(<h1>Page not found</h1>)
val badRequest = BadRequest(views.html.form(formWithErrors))
val oops = InternalServerError("Oops")
val anyStatus = Status(488)("Strange response type")

處理 Redirect

// 303 SEE_OTHER
def index = Action {
  Redirect("/user/home")
}

// 301 MOVED_PERMANENTLY
def index = Action {
  Redirect("/user/home", MOVED_PERMANENTLY)
}

如果只是定義介面,還不實作,可以用 TODO

def index(name:String) = TODO

HTTP Routing

定義 /conf/routes

Play 支援兩種 router: 預設建議使用第一種

  1. dependency injected router
  2. static router

如果要更換 router 為 static router,可以在 build.sbt 中調整設定。如果在 routes 符合兩個以上的路由,那就依照路由宣告的順序使用第一個。

routesGenerator := StaticRoutesGenerator

route 的第一個欄位是填寫 HTTP method,有以下這六種

GET, PATCH, POST, PUT, DELETE, HEAD


route 範例

# 固定路徑
GET   /clients/all          controllers.Clients.list()

# 動態取得路徑中的一部份,可以用 regular expression: [^/]+
GET   /clients/:id          controllers.Clients.show(id: Long)

# *id 就等同於 .+
GET   /files/*name          controllers.Application.download(name)

# 使用自訂的 regular expression,如果要自動轉型,才填寫 id:Long,否則可以省略不寫
GET   /items/$id<[0-9]+>    controllers.Items.show(id: Long)

# 可以給予參數的預設值
GET   /items/               controllers.Items.show(id: Long=10)

Manipulating results

Result 的 Content-Type

// Content-Type: text/plain
val textResult = Ok("Hello World!")

// Content-Type: application/xml
val xmlResult = Ok(<message>Hello World!</message>)

// text/html
val htmlResult = Ok(<h1>Hello World!</h1>).as("text/html")

// text/html; charset=utf-8
val htmlResult2 = Ok(<h1>Hello World!</h1>).as(HTML)

可以增加或修改 HTTP Response Header

val result = Ok("Hello World!").withHeaders(
  CACHE_CONTROL -> "max-age=3600",
  ETAG -> "xx")

設定 Response Cookie

// 增加 cookie: theme
val result = Ok("Hello world").withCookies(
  Cookie("theme", "blue"))

// 清除 cookie: theme
val result2 = result.discardingCookies(DiscardingCookie("theme"))

// 增加 cookie: theme, 同時刪除 cookie: skin
val result3 = result.withCookies(Cookie("theme", "blue")).discardingCookies(DiscardingCookie("skin"))

預設的 charset 為 utf-8,可以修改

利用 scala 的 implicit instance,可以改變 HTML 的 charset

class Application extends Controller {

  implicit val myCustomCharset = Codec.javaSupported("iso-8859-1")

  def index = Action {
    // text/html; charset=iso-8859-1 
    Ok(<h1>Hello World!</h1>).as(HTML)
  }
}

因為 HTML 是用以下的方式定義的

def HTML(implicit codec: Codec) = {
  "text/html; charset=" + codec.charset
}

Session and Flash scopes

如果需要在不同 HTTP requests 之間共用資料,可以儲存在 Session 或是 Flash scopes,Session 會在整個 user Session 都可以使用,Flash scope 只能在下一個 request 中使用。

session 名稱預設為 PLAYSESSION,可以在 application.conf 用 session.cookieName 修改,(為了簡化相容性的問題,其實可以直接改成 JSESSIONID),PLAYSESSION 有使用 private key 加密以避免 client 竄改。

application.conf 的 play.http.session.maxAge (ms) 可設定 session 存活的時間

如果要暫存一些資料,可以在 SESSION 中存入 unique ID ,然後在 play 的 cache 機制中儲存資料


session 的新增與刪除

// 存入 session
Ok("Welcome!").withSession(
  "connected" -> "user@gmail.com")

Ok("Hello World!").withSession(
  request.session + ("saidHello" -> "yes"))

// 移除 session: theme
Ok("Theme reset!").withSession(
  request.session - "theme")

// 讀取 session value,判斷 user 是否已經登入系統
def index = Action { request =>
  request.session.get("connected").map { user =>
    Ok("Hello " + user)
  }.getOrElse {
    Unauthorized("Oops, you are not connected")
  }
}

// 直接用新的 session 將舊的 session 刪除
Ok("Bye").withNewSession

// 取得 flash scope value: success
def index = Action { implicit request =>
  Ok {
    request.flash.get("success").getOrElse("Welcome!")
  }
}

// 新增 flash scope: success
def save = Action {
  Redirect("/home").flashing(
    "success" -> "The item has been created")
}

如果要在畫面上使用 flash scope value: success

@()(implicit flash: Flash)

@flash.get("success").getOrElse("Welcome!")

controller 必須用 implicit instance request 傳送到 views

def index = Action { implicit request =>
  Ok(views.html.index())
}

Body parsers

一般的 http request 資料會暫存記憶體中,可以用 BodyParser 來處理。如果 http request body 內容太長,就需要 以 stream 的方式來處理,以免資料需要整個暫存在記憶體中。

Play 是使用 Akka Strems 處理 stream 資料。


內建的 body parsers 可以處理 JSON, XML, forms

Action 是以 Request[A] => Result 定義的,只要有可以處理 A 的 BodyParser,就可以處理對應的 Request Content

trait Action[A] extends (Request[A] => Result) {
  def parser: BodyParser[A]
}

trait Request[+A] extends RequestHeader {
  def body: A
}

預設會以 request header: Content-Type 作為選用 BodyParser 的條件。例如 Content-Type 為 application/json,就會使用 JsValue 的 BodyParser,Content-Type 為 application/x-form-wwww-urlencoded,就 paring 為 Map[String, Seq[String]]


預設 body parser 會產生 AnyContent 資料型別,然後再以 asJson 轉換成 JsValue

def save = Action { request =>
  val body: AnyContent = request.body
  val jsonBody: Option[JsValue] = body.asJson

  // Expecting json body
  jsonBody.map { json =>
    Ok("Got: " + (json \ "name").as[String])
  }.getOrElse {
    BadRequest("Expecting application/json request body")
  }
}

以下是 default body parser 支援的資料對應

  • text/plain: String, asText
  • application/json: JsValue, asJson
  • application/xml, text/xml 或是 application/XXX+xml: scala.xml.NodeSeq, asXml
  • application/form-url-encoded: Map[String, Seq[String]], asFormUrlEncoded
  • multipart/form-data: MultipartFormData, asMultipartFormData
  • 其他的content: RawBuffer, asRaw

選用適當的 body parser

def save = Action(parse.json) { request =>
  Ok("Got: " + (request.body \ "name").as[String])
}

可以用 curl 測試,會得到 Got: user name

curl 'http://localhost:9000/test' -H 'Content-Type:application/json' --data-binary '{"name": "user name","interval": "month"}'

如果使用 parse.tolerantJson,就會忽略 Content-Type,直接嘗試將 content body 以 json parsing

def save = Action(parse.tolerantJson) { request =>
  Ok("Got: " + (request.body \ "name").as[String])
}

以下例子可以將 request body 儲存到一個檔案中

def save = Action(parse.file(to = new File("/tmp/upload"))) { request =>
  Ok("Saved the request content to " + request.body)
}

storeInUserFile 是一個自訂的 body parser,可以從 session 取得 username,並將 body 以 user 名稱儲存為檔案

val storeInUserFile = parse.using { request =>
  request.session.get("username").map { user =>
    file(to = new File("/tmp/" + user + ".upload"))
  }.getOrElse {
    sys.error("You don't have the right to upload here")
  }
}

def save = Action(storeInUserFile) { request =>
  Ok("Saved the request content to " + request.body)
}

application.conf 中是以這個設定值,決定最大的 content length

play.http.parser.maxMemoryBuffer=128K

可以直接用 maxLength 覆寫這個設定值,parse.maxLength 可以封裝任何一個 body parser

// Accept only 10KB of data.
def save = Action(parse.text(maxLength = 1024 * 10)) { request =>
  Ok("Got: " + text)
}

// Accept only 10KB of data.
def save = Action(parse.maxLength(1024 * 10, storeInUserFile)) { request =>
  Ok("Saved the request content to " + request.body)
}

Custom body parser

note: 這部分我決定先跳過不看,等比較熟悉之後,再去了解

Actions composition

如果需要做一個 Action logging decorator,記錄所有呼叫這個 action 的 request。

首先要實作 invokeBlock method,ActionBuilder 產生的每一個 action 都會呼叫。

import play.api.mvc._

object LoggingAction extends ActionBuilder[Request] {
  def invokeBlock[A](request: Request[A], block: (Request[A]) => Future[Result]) = {
    Logger.info("Calling action")
    block(request)
  }
}

使用剛剛的 LoggingAction

def index = LoggingAction {
  Ok("Hello World")
}

def submit = LoggingAction(parse.text) { request =>
  Ok("Got a body " + request.body.length + " bytes long")
}

用上面的寫法,會需要寫很多個 action buidler,如果要改成將既有的 Action 增加 Logging 的功能,可以用 wrapping actions 的方法實作。

import play.api.mvc._

case class Logging[A](action: Action[A]) extends Action[A] {

  def apply(request: Request[A]): Future[Result] = {
    Logger.info("Calling action")
    action(request)
  }

  lazy val parser = action.parser
}

也可以不產生 action class,直接定義 logging method

import play.api.mvc._

def logging[A](action: Action[A])= Action.async(action.parser) { request =>
  Logger.info("Calling action")
  action(request)
}

Actions 可以利用 composeAction 方法 mixed in to action builder

object LoggingAction extends ActionBuilder[Request] {
  def invokeBlock[A](request: Request[A], block: (Request[A]) => Future[Result]) = {
    block(request)
  }
  override def composeAction[A](action: Action[A]) = new Logging(action)
}

現在就可以這樣使用 LogginAction,或是用 mixin wrapping actions without action builder 的方法

def index = LoggingAction {
  Ok("Hello World")
}

def index = Logging {
  Action {
    Ok("Hello World")
  }
}

這個例子是修改 request,增加一個 remoteAddress 欄位

import play.api.mvc._

def xForwardedFor[A](action: Action[A]) = Action.async(action.parser) { request =>
  val newRequest = request.headers.get("X-Forwarded-For").map { xff =>
    new WrappedRequest[A](request) {
      override def remoteAddress = xff
    }
  } getOrElse request
  action(newRequest)
}

這個例子是拒絕 http 連線,只允許 https 通過

import play.api.mvc._

def onlyHttps[A](action: Action[A]) = Action.async(action.parser) { request =>
  request.headers.get("X-Forwarded-Proto").collect {
    case "https" => action(request)
  } getOrElse {
    Future.successful(Forbidden("Only HTTPS requests allowed"))
  }
}

修改 response,增加一個 response header: X-UA-Compatible

import play.api.mvc._
import play.api.libs.concurrent.Execution.Implicits._

def addUaHeader[A](action: Action[A]) = Action.async(action.parser) { request =>
  action(request).map(_.withHeaders("X-UA-Compatible" -> "Chrome=1"))
}

四個 pre-defined traits,實作了 ActionFunction,用來對不同資料進行處理。我們也可以實作 invokeFunction,實作自訂的 ActionFunction。

  1. ActionTransformer 可修改 request, 增加資料
  2. ActionFilter 選擇性攔截 request,可用來產生 error
  3. ActionRefiner 以上兩種的 general case
  4. ActionBuilder 以 Request 為參數,可以產生新的 actions

使用範例:Authentication

import play.api.mvc._

class UserRequest[A](val username: Option[String], request: Request[A]) extends WrappedRequest[A](request)

object UserAction extends
    ActionBuilder[UserRequest] with ActionTransformer[Request, UserRequest] {
  def transform[A](request: Request[A]) = Future.successful {
    new UserRequest(request.session.get("username"), request)
  }
}

使用範例:增加資料到 request

// add Item to UserRequest
import play.api.mvc._

class ItemRequest[A](val item: Item, request: UserRequest[A]) extends WrappedRequest[A](request) {
  def username = request.username
}

// looks up that item and returns Either an error (Left) or a new ItemRequest (Right)
def ItemAction(itemId: String) = new ActionRefiner[UserRequest, ItemRequest] {
  def refine[A](input: UserRequest[A]) = Future.successful {
    ItemDao.findById(itemId)
      .map(new ItemRequest(_, input))
      .toRight(NotFound)
  }
}

使用範例: Validating requests

object PermissionCheckAction extends ActionFilter[ItemRequest] {
  def filter[A](input: ItemRequest[A]) = Future.successful {
    if (!input.item.accessibleByUser(input.username))
      Some(Forbidden)
    else
      None
  }
}

利用 ActionBuilder,以及 andThen,把以上的 action function 合併在一起

def tagItem(itemId: String, tag: String) =
  (UserAction andThen ItemAction(itemId) andThen PermissionCheckAction) { request =>
    request.item.addTag(tag)
    Ok("User " + request.username + " tagged " + request.item.id)
  }

Content negotiation

這個機制可以用相同的 URI 處理不同的 content boday,例如同一個 URI 網址可以同時接受 XML, JSON。也可以利用 Accept-Language 欄位,決定回傳的 response 要用什麼語系的內容。

play.api.mvc.RequestHeader#acceptLanguages 可取得 Accept-Language 的資料

play.api.mvc.RequestHeader#acceptedTypes 可取得可接受的 request MIME types 列表,也就是 Accept 欄位。實際上 Accept 欄位並沒有確切的 MIME type,而是 media range 例如 text/*, /,controller 提供 render method,處理 media range。

val list = Action { implicit request =>
  val items = Item.findAll
  render {
    case Accepts.Html() => Ok(views.html.list(items))
    case Accepts.Json() => Ok(Json.toJson(items))
  }
}

可以使用 play.api.mvc.Accepting case class 產生特定 MIME type 的自訂 extractor。

  val AcceptsMp3 = Accepting("audio/mp3")
  render {
    case AcceptsMp3() => ???
  }

Handling errors

HTTP application 的 error 主要有兩種:client errors, server errors。

Play 會自動偵測 client errors,包含 malformed header value, unsupported content types, requests for unknown resource。也會自動處理一些 server errors,例如 code throws an exception,Play 會自動產生 error page。

以下為 custom error handler,實作了兩個 methods: onClientError 及 onServerError。

import play.api.http.HttpErrorHandler
import play.api.mvc._
import play.api.mvc.Results._
import scala.concurrent._
import javax.inject.Singleton;

@Singleton
class ErrorHandler extends HttpErrorHandler {

  def onClientError(request: RequestHeader, statusCode: Int, message: String) = {
    Future.successful(
      Status(statusCode)("A client error occurred: " + message)
    )
  }

  def onServerError(request: RequestHeader, exception: Throwable) = {
    Future.successful(
      InternalServerError("A server error occurred: " + exception.getMessage)
    )
  }
}

可在 application.conf 設定 error handler

play.http.errorHandler = "com.example.ErrorHandler"

也可以利用 default error handler 延伸自己的功能。以下就是不改變 development error message,並修改特定的 onForbidden error page。

import javax.inject._

import play.api.http.DefaultHttpErrorHandler
import play.api._
import play.api.mvc._
import play.api.mvc.Results._
import play.api.routing.Router
import scala.concurrent._

@Singleton
class ErrorHandler @Inject() (
    env: Environment,
    config: Configuration,
    sourceMapper: OptionalSourceMapper,
    router: Provider[Router]
  ) extends DefaultHttpErrorHandler(env, config, sourceMapper, router) {

  override def onProdServerError(request: RequestHeader, exception: UsefulException) = {
    Future.successful(
      InternalServerError("A server error occurred: " + exception.getMessage)
    )
  }

  override def onForbidden(request: RequestHeader, message: String) = {
    Future.successful(
      Forbidden("You're not allowed to access this resource.")
    )
  }
}

References

HTTP programming