Release Notes

Discover the new features, updates, and known limitations in this release of the DC/OS Apache Spark Service

Release Notes for DC/OS Apache Spark Service version 2.6.0-2.3.2

New features

  • Upgraded Spark and Spark History Server to 2.3.2
  • Added DC/OS Spark CLI support for --jars
  • Added CNI Support for Dispatcher, Driver, and Executors for Docker and Mesos containerizers
  • Added CNI labels support for Mesos containerizer
  • Added package configuration for CNI:
    • virtual_network_enabled
    • virtual_network_name
    • virtual_network_plugin_labels
  • Spark Dispatcher by default launches Spark Drivers and Executors in the same virtual network it was launched in itself
NOTE: Limitations of current Spark CNI support:
  • Configuration of network plugin labels from DC/OS UI supported only in JSON editing mode
  • Network plugin labels are not supported by Docker containerizer
  • Currently, DC/OS AdminRouter doesn't support virtual networks so DC/OS {{ model.techShortName }} endpoints will not be accessible from CLI, and jobs need to be submitted from a routable network

Updates

  • SPARK_HOME environment variable defaults to /opt/spark in Dockerfile and executable scripts
  • Switched to Spark’s own StatsD Sink instead of 3rd-party dependency
  • Updated dcos-commons bootstrap version to 0.55.2

Bug fixes

  • Fixed bug for Dispatcher restarting duplicate Spark drivers after agent restart in --supervised mode
  • Fixed bug for CLI incorrect --jars parsing resulting in submit failure