How to manage `steps` in Amazon EMR

Problem: we submit steps with aws emr command, and then we discovered that the step was failed. How can we trace back the error?

Outline:

  1. Copy the Hive script into S3
  2. Run with AWS CLI
  3. Check for the log in Amazon EMR

1. Copy the Hive script into S3

Suppose you have a script like this, and you would like to run it on AWS EMR.

# File: 07-CgiEventCount.q
CREATE EXTERNAL TABLE IF NOT EXISTS found_cgi_event_count (
cgi STRING,
eventCount INT)
COMMENT 'Here we only deal with CGI that are found in the map. And the total number of CDR events for that CGI'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LOCATION '${INPUT_RADIX}/found_cgi_event_count';

INSERT OVERWRITE TABLE found_cgi_event_count
SELECT cgi, count(*)
FROM mobility_data_aggregated_by_user_found_cgi_hour_all
GROUP BY cgi
SORT BY cgi;

I like to put all the scripts into 1 folder in S3. And I maintain the same copy of that folder in my local machine. By that, I can edit the script locally if something goes wrong and push the new version of the script into the S3_SCRIPT_PATH

The syntax for copying local file to S3:

aws s3 07-CgiEventCount.q s3://bucket-name/quynh/scripts

Run with AWS CLI

Sample command:

# In your local shell
aws emr add-steps --cluster-id $CLUSTER_ID --steps Type=HIVE,Name="CGI that are found in the map. And the total number of CDR events for that CGI in each hour.",ActionOnFailure=CONTINUE,Args=[-f,$S3_SCRIPT_PATH/07-CgiEventCount.q,-d,INPUT_RADIX=$S3_HIVE_TABLE_PATH]

The INPUT_RADIX variable will replace the content of that same variable in 07-CgiEventCount.q script.

This is the output from AWS CLI:

{
    "StepIds": [
        "s-Y3ARB4KF88XV"
    ]
}

Pay attention to the StepIds, this is useful when to you want to check for the status of the step. Some of the command that I use frequently is:

# In your local shell
CLUSTER_ID=<your cluster id>
STEP_ID=<your step id>

aws emr describe-step --cluster-id $CLUSTER_ID --step-id $STEP_ID

This one is to list all the pending | running steps:

aws emr list-steps --cluster-id $CLUSTER_ID --step-states RUNNING PENDING

Check for stdout, stderr in Amazon EMR

You ssh into the Amazon EMR, and then all the log for different steps submitted through aws emr resides in the folder: /mnt/var/log/hadoop/steps. Check out an example below for script to recursively retrieve new content from file stderr

# Inside the cluster
cd /mnt/var/log/hadoop/steps/s-Y3ARB4KF88XV
tail -f stderr

# or
tail -f stdout
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s