Missing “=” sign

I was struggling with this piece of code. I have one similar piece of code, and it was running perfectly. I have to go back and forth, delete gradually pieces and pieces of code to compare between the two.

Turn out that the mistake is really really small, as shown in line 38.


The problem I got:

Name: org.apache.spark.SparkException
Message: Job aborted due to stage failure: Task 0 in stage 365.0 failed 4 times, most recent failure: Lost task 0.3 in stage 365.0 (TID 12524, lamar.homenet.telecomitalia.it): java.lang.ClassCastException

import org.apache.spark.annotation.Experimental
import org.apache.spark.sql.{ Column, Row }
import org.apache.spark.sql.catalyst.expressions.aggregate.{ AggregateExpression, Complete }
import org.apache.spark.sql.execution.aggregate.ScalaUDAF
import org.apache.spark.sql.types.{ StructType, DataType, IntegerType, StructField, StringType }

class CustomTower extends UserDefinedAggregateFunction {
// Input Data Type Schema
def inputSchema: StructType = StructType(Array(StructField("item", StringType)))

def bufferSchema = StructType(Array(
StructField("cnt", IntegerType),
StructField("list", StringType)))

// Returned Data Type
def dataType: DataType = IntegerType

// Self-explaining
def deterministic = true

// This function is called whenever key changes
def initialize(buffer: MutableAggregationBuffer) = {
buffer(0) = -1
buffer(1) = ""

// Iterate over each entry of a group
def update(buffer: MutableAggregationBuffer, input: Row) = {

buffer(0) = buffer.getInt(0) + 1

// Merge two partial aggregates
def merge(buffer1: MutableAggregationBuffer, buffer2: Row) = {
buffer1(0) = buffer1.getInt(0) + 1

def evaluate(buffer: Row) = {

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s