速博网上娱乐


速博网上娱乐闻见芬芳
速博网上娱乐中午的烈日是那么的毒辣,花草耷拉下了脑袋,鸟儿不见了影,连禅也没了声。就我父亲而言他是不太喜欢蝉的因为蝉太吵了。
速博网上娱乐有一天
速博网上娱乐为你作一个温暖的港湾
速博网上娱乐驰骋疆场数十载,心念还家,
速博网上娱乐快乐

皇冠足球比分

大发论坛时时彩平台尊龙娱乐是黑网吗海天国际娱乐城注册送钱凯发娱乐亚美国际娱乐城 金龙国际 www.am8.com 亚洲城娱乐 老k国际娱乐城 666k8.com 金威国际娱乐城 名人国际娱乐城 ag娱乐平台 V博娱乐城 注册送10元无需申请注册送钱e8889.com盈禾娱乐城百利宫大发8注册送钱e8889.com顶级在线娱乐城赌球什么方法最稳赚凯时娱乐注册送钱e8889.com威尼斯娱乐城奔驰在线娱乐导航

Spark Job Scheduling

最近由于项目需要在研究spark相关的内容,形成了一些技术性文档,发布这记录下,懒得翻译了。

   There are some spaces the official documents didn't explain very clearly, especially on some details. Here are given some more explanations base on the practices  I did  and the source codes I read these days.

(The official document link is http://spark.apache.org/docs/latest/job-scheduling.html)

  1. There are two different schedulers in current spark implementation, FIFO is the default setting and the initial way that spark implement.
  2. Both FIFO and FAIR schedulers can support the basic functionality that multiple parallel jobs run simultaneously, the prerequisite is that they are submitted from separate threads. (i.e., in single thread, the jobs are executed in order)
  3. In FIFO Scheduler, the jobs which are submitted earlier has higher priority and possibility than those later jobs. But it doesn't mean that the first job will execute first, it is also possible that later jobs run before the earlier ones if the resources of the whole cluster are not occupied. However,  the FIFO scheduler will cause the worst case: if the first jobs are large, the later jobs maybe suffer significant delay.
  4. The FAIR Scheduler is the way corresponding to Hadoop FAIR scheduler and enhancement for FIFO. In FIFO fashion, there is only one factor Priority will be considered in SchedulableQueue; While in FAIR fashion, more factors will be considered including minshare, runningtasks, weight (You can reference the code below if interest).Similarly, the jobs don't always run by following the rules by FairSchedulingAlgorithm strictly, while as a whole, the FAIR scheduler really alleviate largely the delay time for small jobs by adjusting the parameters which were delayed significantly in FIFO fashion in my observation through the concurrent JMeter tests。

      

private[spark] class FIFOSchedulingAlgorithm extends SchedulingAlgorithm {
  override def comparator(s1: Schedulable, s2: Schedulable): Boolean = {
    val priority1 = s1.priority
    val priority2 = s2.priority
    var res = math.signum(priority1 - priority2)
    if (res == 0) {
      val stageId1 = s1.stageId
      val stageId2 = s2.stageId
      res = math.signum(stageId1 - stageId2)
    }
    if (res < 0) {
      true
    } else {
      false
    }
  }
}

private[spark] class FairSchedulingAlgorithm extends SchedulingAlgorithm {
  override def comparator(s1: Schedulable, s2: Schedulable): Boolean = {
    val minShare1 = s1.minShare
    val minShare2 = s2.minShare
    val runningTasks1 = s1.runningTasks
    val runningTasks2 = s2.runningTasks
    val s1Needy = runningTasks1 < minShare1
    val s2Needy = runningTasks2 < minShare2
    val minShareRatio1 = runningTasks1.toDouble / math.max(minShare1, 1.0).toDouble
    val minShareRatio2 = runningTasks2.toDouble / math.max(minShare2, 1.0).toDouble
    val taskToWeightRatio1 = runningTasks1.toDouble / s1.weight.toDouble
    val taskToWeightRatio2 = runningTasks2.toDouble / s2.weight.toDouble
    var compare: Int = 0

    if (s1Needy && !s2Needy) {
      return true
    } else if (!s1Needy && s2Needy) {
      return false
    } else if (s1Needy && s2Needy) {
      compare = minShareRatio1.compareTo(minShareRatio2)
    } else {
      compare = taskToWeightRatio1.compareTo(taskToWeightRatio2)
    }

    if (compare < 0) {
      true
    } else if (compare > 0) {
      false
    } else {
      s1.name < s2.name
    }
  }
 


      

  5.The pools in FIFO and FAIR schedulers

      

 

posted @ 2015-11-24 12:53 踏学寻梅 阅读(...) 评论(...) 编辑 收藏