Introduction to the Course’s PLs

Programming Languages for Big Data Processing

Scala and Python

The de facto languages of Big Data and data science are

• Scala Mostly used for data intensive systems
• Python Mostly used for data analytics tasks

Other languages include

• Java The “assembly” of big data systems; the language that most big data infrastructure is written into.
• R The statistician’s tool of choice. Great selection of libraries for serious data analytics, great plotting tools.

In our course, we will be using Scala.

Scala and Python from 10k feet

• Both support object orientation, functional programming and imperative programming
  • Scala’s strong point is the combination of FP and OO
  • Python’s strong point is the combination of OO and IP
• Python is interpreted, Scala is compiled

Hello world

Scala

object Hello extends App {
    println("Hello, world")
    for (i <- 1 to 10) {
      System.out.println("Hello")
    }
}

• Scala is compiled to JVM bytecode
• Can interoperate with JVM libraries
• Scala is not sensitive to spaces/tabs. Blocks are denoted by { and }

Python

#!/usr/bin/env python3

for i in range(1, 10):
  print("Hello, world")

• Python is interpreted
• Python is indentation sensitive: blocks are denoted by a TAB or 2 spaces.

Declarations

Scala

val a: Int = 5
val b = 5
b = 6 // re-assignment to val

// Type of foo is inferred
val foo = new ImportantClass(...)

var a = "Foo"
a = "Bar"
a = 4 // type mismatch

• Type inference used extensively
• Two types of variables: vals are single-assignment, vars are multiple assignment

Python

a : int = 5
a = "Foo"

a = ImportantClass(...)

• Optional typing, not enforced at runtime

Declaring functions

Scala

def max(x: Int, y: Int): Int = 
  if (x >= y) x else y

• Statically typed
• Evaluated expressions have types
• The return type is the most generic type of all return expressions

Python

def max(x : int, y : int) -> int:
  if x >= y:
    return x
  else:
    return y

• Python is dynamically typed
• Mostly based on statements
• Types are optional

Higher order functions

Scala

def bigger(x: Int, y: Int,
  f: (Int,Int) => Boolean) =
  f(x, y)

bigger (1, 2, (x, y) => (x < y))
bigger (1, 2, (x, y) => (x > y))
// Compile error
bigger (1, 2, x => x)

Python

def bigger(x, y, f):
  return f(x, y)

bigger(1,2, lambda x,y: x > y)
bigger(1,2, lambda x,y: x < y)
# Runtime error
bigger(1,2, lambda x: x)

bigger is a higher-order function, i.e. a function whose behaviour is parametrised by another function. f a function parameter. To call a HO function, we need to pass a function with the appropriate argument types. The compiler checks this in the case of Scala.

Declaring classes

Scala

class Foo(val x: Int,
          var y: Double = 0.0)

// Type of a is inferred
val a = new Foo(1, 4.0)
println(a.x) //x is read-only
println(a.y) //y is read-write
a.y = 10.0
println(a.y) //y is read-write
a.y = "Foo"   // Type mismatch, y is double

• val means a read-only attribute. var is read-write
• A default constructor is created automatically

Python

class Foo():
  def __init__(self, x, y):
    self.x = x
    self.y = y

a = Foo(3,2)
print a.x
a.x = "foo"
print a.x

• Attributes are declared in the constructor
• Attributes are by default read-write
• No types are enforced

Object-Oriented programming

Scala

class Foo(val x: Int,
          var y: Double = 0.0)

class Bar(x: Int, y: Int, z: Int)
  extends Foo(x, y)

trait Printable {
  val s: String
  def asString() : String
}

class Baz(x: Int, y: Double, private val z: Int)
  extends Foo(x, y) with Printable {
  override val s: String = s
  override def asString(): String = ???
}

Python

class Foo():
  def __init__(self, x, y):
    self.x = x
    self.y = y

class Bar(Foo):
  def __init__(self, x, y, z):
    Foo.__init__(self, x, y)
    self.z = z

In both cases, the traditional rules of method overriding apply. Traits in Scala are similar to default interfaces in Java > 9; in addition, they can include attributes (state).

Data classes

Scala

case class Address(street: String, 
  number: Int)
case class Person(name: String, 
  address: Address)

val p = Person("G", Address("a", 2))

Python >= 3.7

from dataclasses import dataclass

@dataclass
class Address:
    street: str
    number: int

@dataclass
class Person:
    name: str
    addr: Address

p = Person("G", Address("a", 2))

Data classes are blueprints for immutable objects. We use them to represent data records. Both languages implement equals (or __eq__) for them, so we can compare objects directly.

Pattern matching in Scala

Pattern matching is if..else on steroids

// Code for demo only, won't compile

value match {
  // Match on a value, like if
  case 1 => "One"
  // Match on the contents of a list
  case x :: xs => "The remaining contents are " + xs
  // Match on a case class, extract values
  case Email(addr, title, _) => s"New email: $title..."
  // Match on the type
  case xs : List[_] => "This is a list"
  // With a pattern guard
  case xs : List[Int] if xs.head == 5 => "This is a list of integers"
  case _ => "This is the default case"
}

Reading ahead

This is by far not an introduction to either programming languages. Please read more here

• Scala documentation
• Python documentation

Bibliography

[1]

G. Hutton, “A tutorial on the universality and expressiveness of fold,” Journal of Functional Programming, vol. 9, no. 4, pp. 355–372, 1999.

Copyright

This work is (c) 2017, 2018, 2019, 2020, 2021 - onwards by TU Delft and Georgios Gousios and licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.