Demystifying Amazon SageMaker Training for scikit-learn Lovers

aws

sagemaker

How to build, train, and deploy a machine learning model with Amazon SageMaker.

Published

June 8, 2022

Enviornment

This notebook is prepared with Amazon SageMaker Studio using Python 3 (Data Science) Kernel and ml.t3.medium instance.

About

This post is about simplifying Amazon SageMaker end-to-end machine learning workflow for newcomers. We will go slowly and try to understand each step involved in training and deploying a model. As an example, we will take Kaggle Boston Housing dataset and train a SageMaker built-in algorithm Linear Learner on it. At the end of this tutorial, you will know how to use Amazon SageMaker to build, train, and deploy a machine learning model.

Introduction

A typical SageMaker machine learning flow has the following five steps. If you have a good understanding of them then you can use this approach to train any other model with SageMaker. 1. Put data on S3 bucket
In most use cases, you will keep your training data on S3 bucket. You may also require to preprocess your data and for this you can use SageMaker Data Wrangler. In this post, we will consider that data has already been processed and will upload it to S3 for training. 2. Configure the training job
While configuring a training job you need to take care of the following requirements
a. select the algorithm you want to use for your training
b. set the hyperparameters (if any)
c. define the infrastructure requirements like how many CPUs or GPUs you want to throw at your training run 3. Launch training job
Here we will tell the training job where the input data is located, and where should the output artifacts be stored once the training is done. With this setting, we are ready to start the training run. Once the training is started SageMaker will automatically provision the required infrastructure for our run, and when the training is complete it will be terminated, and you will only be billed for what you have used. 4. Deploy model and make predictions
Deploy the model to make real-time predictions. Again, you need to define the infrastructure requirements where you want your model to be deployed. 5. Clean Up (Optional)
If you are experimenting, you may want to terminate the machine on which you have deployed your model to avoid any unnecessary charges.

Put Data on S3 Bucket

Reading and Checking the Data

In this post, we will be using Boston Housing Dataset. This is a small dataset with 506 rows and 14 columns. medv is the target variable which means median value of owner-occupied homes in $1000s. This dataset is also available with this notebook. Let’s read it and see how it looks.

import pandas as pd
import numpy as np

data_location = "./datasets/2022-06-08-sagemaker-training-overview/"

df = pd.read_csv(data_location + 'housing.csv')
df.head()

	crim	zn	indus	nox	age	rm	dis	rad	tax	ptratio	lstat	medv
0	0.00632	18.0	2.31	0.538	6.575	65.2	4.0900	1	296.0	15.3	4.98	24.0
1	0.02731	0.0	7.07	0.469	6.421	78.9	4.9671	2	242.0	17.8	9.14	21.6
2	0.02729	0.0	7.07	0.469	7.185	61.1	4.9671	2	242.0	17.8	4.03	34.7
3	0.03237	0.0	2.18	0.458	6.998	45.8	6.0622	3	222.0	18.7	2.94	33.4
4	0.06905	0.0	2.18	0.458	7.147	54.2	6.0622	3	222.0	18.7	5.33	36.2

let’s quickly check the dimensions of our loaded dataset.

df.shape

(506, 13)

The good thing about this dataset is that it does not require any preprocessing as all the features are already in numerical format (no categorical features), and also there are no missing values. We can quickly verify these assumptions.

Check the feature data types.

df.dtypes

crim       float64
zn         float64
indus      float64
chas         int64
nox        float64
age        float64
rm         float64
dis        float64
rad          int64
tax        float64
ptratio    float64
lstat      float64
medv       float64
dtype: object

Check if there are any missing values in our dataset.

df.isnull().values.any()

False

Preparing the Data

At this point, our data is ready to be used for training but we also need to check the algorithm we want to use for any specific requirements. We have selected Linear Learner so let’s check its documentation: Linear Learner Algorithm

In the documentation it says >For training, the linear learner algorithm supports both recordIO-wrapped protobuf and CSV formats. For the application/x-recordio-protobuf input type, only Float32 tensors are supported. For the text/csv input type, the first column is assumed to be the label, which is the target variable for prediction. You can use either File mode or Pipe mode to train linear learner models on data that is formatted as recordIO-wrapped-protobuf or as CSV.

It means that we can use CSV format for our training data. It also mentions that the first column in the training dataset should be the target label. So let’s move our target label medv to the first column.

df = pd.concat([df['medv'], df.drop('medv', axis='columns')], axis='columns')

df.head()

	medv	crim	zn	indus	nox	age	rm	dis	rad	tax	ptratio	lstat
0	24.0	0.00632	18.0	2.31	0.538	6.575	65.2	4.0900	1	296.0	15.3	4.98
1	21.6	0.02731	0.0	7.07	0.469	6.421	78.9	4.9671	2	242.0	17.8	9.14
2	34.7	0.02729	0.0	7.07	0.469	7.185	61.1	4.9671	2	242.0	17.8	4.03
3	33.4	0.03237	0.0	2.18	0.458	6.998	45.8	6.0622	3	222.0	18.7	2.94
4	36.2	0.06905	0.0	2.18	0.458	7.147	54.2	6.0622	3	222.0	18.7	5.33

We can now proceed with splitting our data into train and validation sets. After splitting we will export it as CSV files so it could be uploaded to S3 in the next section. Before exporting our dataframe into any format we should also check SageMaker general instructions for that format. For this use the link Common Data Formats for Training and check section Using CSV Format. Here it mentions

To use data in CSV format for training, in the input data channel specification, specify text/csv as the ContentType. Amazon SageMaker requires that a CSV file does not have a header record and that the target variable is in the first column.

We have already moved our target label to the first column, and while exporting our dataframe to CSV format we should omit column headers as well.

from sklearn.model_selection import train_test_split

# out data size is very small so we will use a small test set
training_data, validation_data = train_test_split(df, test_size=0.1, random_state=42)

training_data.to_csv(data_location + "training_data.csv", index=False, header=False)
validation_data.to_csv(data_location + "validation_data.csv", index=False, header=False)

The next step is to upload this data to S3 bucket, and for this we will take the help of SageMaker Python SDK. There are two Python SDKs (Software Development Kit) available for SageMaker.

AWS SDK for Python (Boto3). It provides low-level access to SageMaker APIs.
SageMaker Python SDK. It provides a high-level API interface, and you can do more with fewer lines of code. Internally it is calling Boto3 APIs.

We will be using SageMaker Python SDK for this post, and you will see that it has an interface similar to scikit-learn, and is a more natural choice for Data Scientists. SageMaker Python SDK documentation is super helpful, and it provides many examples to understand the working of its interface. Make sure that you check it out as well sagemaker.readthedocs.io. If you don’t have much time then I would suggest at least read the following sections from the documentation as we will be using them in the coming sections.

Since we are already running this notebook from SageMaker environment, we don’t need to care about credentials and permissions. We can simply start our new session with SageMaker environment using its SDK.

import sagemaker

session = sagemaker.Session()
role = sagemaker.get_execution_role()
bucket = session.default_bucket()
region = session.boto_region_name

print(f"sagemaker.__version__: {sagemaker.__version__}")
print(f"Session: {session}")
print(f"Role: {role}")
print(f"Bucket: {bucket}")
print(f"Region: {region}")

sagemaker.__version__: 2.88.1
Session: <sagemaker.session.Session object at 0x7ff91637cd50>
Role: arn:aws:iam::801598032724:role/service-role/AmazonSageMaker-ExecutionRole-20220516T161743
Bucket: sagemaker-us-east-1-801598032724
Region: us-east-1

What we have done is * imported the SageMaker Python SDK into our runtime * get a session to work with SageMaker API and other AWS services * get the execution role associated with the user profile. It is the same profile that is available to the user to work from console UI and has AmazonSageMakerFullAccess policy attached to it. * create or get a default bucket to use and return its name. Default bucket name has the format sagemaker-{region}-{account_id}. If it doesn’t exist then our session will automatically create it. You may also use any other bucket in its place given that you have enough permission for reading and writing. * get the region name attached to our session

Next, we will use this session to upload data to our default bucket.

##
# You may choose any other prefix for your bucket. All the data related to this post will be under this prefix.
bucket_prefix = '2022-06-08-sagemaker-training-overview'

Let’s upload our training data first. In the output, we will get the complete path (S3 URI) for our uploaded data.

s3_train_data_path = session.upload_data(
    path=data_location + "training_data.csv",
    bucket=bucket,
    key_prefix=bucket_prefix + '/input/training'
)

print(s3_train_data_path)

s3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/input/training/training_data.csv

Let’s do the same for our validation data.

s3_validation_data_path =  session.upload_data(
    path=data_location + "validation_data.csv",
    bucket=bucket,
    key_prefix=bucket_prefix + '/input/validation_data'
)

print(s3_validation_data_path)

s3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/input/validation_data/validation_data.csv

At this point we have our data available on S3 bucket. We can now proceed to the next step and configure our training job.

Configure the Training Job

In this section, we will first retrieve the Docker container that is relevant to our training algorithm. Then we will create “sagemaker.estimator.Estimator” class object. This estimator object provides a high-level API interface to control end-to-end SageMaker training and deployment tasks. From the estimator, we will also define our infrastructure and hyperparameter tuning requirements. So let’s get started.

Finding the Right Docker Container

AWS SageMaker built-in algorithms are fully managed containers that can be accessed with one call. Each algorithm has a separate container and is also dependent on the region in which you want to run it. Getting the container URI is not a problem as long as we know about the region and the algorithm framework name. We already have the region name from our SageMaker session. To get the algorithm framework name (for linear learner) visit the AWS Docker Registry Paths page. From this page select your region. In my case it is us-east-1. On the regional docker registry page find the algorithm you want to use; Linear Learner in our case. This will give you the example code and algorithm framework name as shown below.

So let’s use the provided sample code to get the container URI for our linear learner algorithm.

from sagemaker import image_uris

image_uri = image_uris.retrieve(framework='linear-learner',region=region)

print(image_uri)

382416733822.dkr.ecr.us-east-1.amazonaws.com/linear-learner:1

Configure the Estimator Object

To configure our estimator we need to fulfill the following requirements. * output path Define the output path where we want to store the trained model artifacts * instance type Since we have a small dataset and not so complex model so a small machine should suffice. ‘ml.m5.large’ will do. It is a general purpose instance with 2vCPU and 8GiB RAM. * hyperparameters For the hyperparameters check the Linear Learner model documentaion. From the documentation, we find that the most important parameters for our problem are * predictor_type: which should be ‘regressor’ in our case * mini_batch_size: default is 1000 which is too large for our small dataset. Let’s use 30 instead.

It is also important to note that the Estimator class will automatically provision a separate ml.m5.large machine to start the training run. This machine will be different from the one on which we are running this Jupyter notebook. Once training is complete this new machine will be terminated and we will be billed for only the time we have used it. This approach makes SageMaker very cost effective. We can keep using small less powerful machines for running Jupyter notebooks, and for training and other heavy workloads, we can provision separate machines for short durations and avoid any unnecessary bills.

##
# define the output path to store trained model artifacts
s3_output_path = f"s3://{bucket}/{bucket_prefix}/output/"

print(s3_output_path)

s3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/output/

from sagemaker.estimator import Estimator

ll_estimator = Estimator(
    image_uri = image_uri, # algorithm container
    role = role, # execution role with necessary permissions
    instance_count = 1,
    instance_type = 'ml.m5.large',
    sagemaker_session = session, # SageMaker API session
    output_path = s3_output_path, # training artifacts output path
    hyperparameters = {
        'predictor_type': 'regressor',
        'mini_batch_size': 30
    }
)

In the above cell, we have defined the hyperparameters within the Estimator object constructor. There is a second way to pass the hyperparameters to the Estimator object using the ‘set_hyperparameters’ function call. This method can be useful when we have a large number of parameters to set, or when we want to change them in multiple training runs.

ll_estimator.set_hyperparameters(
    predictor_type='regressor', 
    mini_batch_size=30)

You might ask that for our problem even a small ml.t3.medium or ml.c5.large machine would have been sufficient. Why have not we used them? The answer to this is that AWS SageMaker at this time supports a limited number of machine types for training jobs and both of them are not supported. If you configure the Estimator object for these instance types you will get an error shown below

An error occurred (ValidationException) when calling the CreateTrainingJob operation: 1 validation error detected: Value 'ml.t3.medium' at 'resourceConfig.instanceType' failed to satisfy constraint: Member must satisfy enum value set: 

[ml.p2.xlarge, ml.m5.4xlarge, ml.m4.16xlarge, ml.p4d.24xlarge, ml.g5.2xlarge, ml.c5n.xlarge, ml.p3.16xlarge, ml.m5.large, ml.p2.16xlarge, ml.g5.4xlarge, ml.c4.2xlarge, ml.c5.2xlarge, ml.c4.4xlarge, ml.g5.8xlarge, ml.c5.4xlarge, ml.c5n.18xlarge, ml.g4dn.xlarge, ml.g4dn.12xlarge, ml.c4.8xlarge, ml.g4dn.2xlarge, ml.c5.9xlarge, ml.g4dn.4xlarge, ml.c5.xlarge, ml.g4dn.16xlarge, ml.c4.xlarge, ml.g4dn.8xlarge, ml.g5.xlarge, ml.c5n.2xlarge, ml.g5.12xlarge, ml.g5.24xlarge, ml.c5n.4xlarge, ml.c5.18xlarge, ml.p3dn.24xlarge, ml.g5.48xlarge, ml.g5.16xlarge, ml.p3.2xlarge, ml.m5.xlarge, ml.m4.10xlarge, ml.c5n.9xlarge, ml.m5.12xlarge, ml.m4.xlarge, ml.m5.24xlarge, ml.m4.2xlarge, ml.p2.8xlarge, ml.m5.2xlarge, ml.p3.8xlarge, ml.m4.4xlarge]

Launch Training Job

To start our training run we need to associate our Estimator object with the training data. Our data is available on S3 bucket but we also need to tell our Estimator object in which format this data is provided. Is it in CSV format? Is it compressed or not? For these data format issues, we can use Input Channels. An input channel is configurations for S3 and file system data sources for AWS SageMaker (check the input docs). All SageMaker built-in algorithms require at least one training input channel, and more can be passed for validation and testing. In our case, we have two channels and both are in CSV format. Let’s configure them.

from sagemaker.session import TrainingInput

train_input = TrainingInput(s3_train_data_path, content_type="text/csv")
validation_input = TrainingInput(s3_validation_data_path, content_type="text/csv")


ll_data = {
    'train': train_input,
    'validation': validation_input
}

Make sure that you use content_type text/csv. Only providing csv will not work and you will get the exception

Error for Training job linear-learner-2022-06-15-07-58-01-908: Failed. Reason: ClientError: No iterator has been registered for ContentType ('csv', '1.0'), exit code: 2

Alright, we are ready. To start the training run call estimator fit function and pass the data input channels. You can read more about fit call from docs sagemaker.estimator.Estimator.fit

#collapse-output
ll_estimator.fit(ll_data)

2022-06-16 09:04:57 Starting - Starting the training job...
2022-06-16 09:05:23 Starting - Preparing the instances for trainingProfilerReport-1655370297: InProgress
.........
2022-06-16 09:06:48 Downloading - Downloading input data......
2022-06-16 09:07:44 Training - Downloading the training image...
2022-06-16 09:08:24 Training - Training image download completed. Training in progress..Docker entrypoint called with argument(s): train
Running default environment configuration script
[06/16/2022 09:08:32 INFO 139637139089216] Reading default configuration from /opt/amazon/lib/python3.7/site-packages/algorithm/resources/default-input.json: {'mini_batch_size': '1000', 'epochs': '15', 'feature_dim': 'auto', 'use_bias': 'true', 'binary_classifier_model_selection_criteria': 'accuracy', 'f_beta': '1.0', 'target_recall': '0.8', 'target_precision': '0.8', 'num_models': 'auto', 'num_calibration_samples': '10000000', 'init_method': 'uniform', 'init_scale': '0.07', 'init_sigma': '0.01', 'init_bias': '0.0', 'optimizer': 'auto', 'loss': 'auto', 'margin': '1.0', 'quantile': '0.5', 'loss_insensitivity': '0.01', 'huber_delta': '1.0', 'num_classes': '1', 'accuracy_top_k': '3', 'wd': 'auto', 'l1': 'auto', 'momentum': 'auto', 'learning_rate': 'auto', 'beta_1': 'auto', 'beta_2': 'auto', 'bias_lr_mult': 'auto', 'bias_wd_mult': 'auto', 'use_lr_scheduler': 'true', 'lr_scheduler_step': 'auto', 'lr_scheduler_factor': 'auto', 'lr_scheduler_minimum_lr': 'auto', 'positive_example_weight_mult': '1.0', 'balance_multiclass_weights': 'false', 'normalize_data': 'true', 'normalize_label': 'auto', 'unbias_data': 'auto', 'unbias_label': 'auto', 'num_point_for_scaler': '10000', '_kvstore': 'auto', '_num_gpus': 'auto', '_num_kv_servers': 'auto', '_log_level': 'info', '_tuning_objective_metric': '', 'early_stopping_patience': '3', 'early_stopping_tolerance': '0.001', '_enable_profiler': 'false'}
[06/16/2022 09:08:32 INFO 139637139089216] Merging with provided configuration from /opt/ml/input/config/hyperparameters.json: {'mini_batch_size': '30', 'predictor_type': 'regressor'}
[06/16/2022 09:08:32 INFO 139637139089216] Final configuration: {'mini_batch_size': '30', 'epochs': '15', 'feature_dim': 'auto', 'use_bias': 'true', 'binary_classifier_model_selection_criteria': 'accuracy', 'f_beta': '1.0', 'target_recall': '0.8', 'target_precision': '0.8', 'num_models': 'auto', 'num_calibration_samples': '10000000', 'init_method': 'uniform', 'init_scale': '0.07', 'init_sigma': '0.01', 'init_bias': '0.0', 'optimizer': 'auto', 'loss': 'auto', 'margin': '1.0', 'quantile': '0.5', 'loss_insensitivity': '0.01', 'huber_delta': '1.0', 'num_classes': '1', 'accuracy_top_k': '3', 'wd': 'auto', 'l1': 'auto', 'momentum': 'auto', 'learning_rate': 'auto', 'beta_1': 'auto', 'beta_2': 'auto', 'bias_lr_mult': 'auto', 'bias_wd_mult': 'auto', 'use_lr_scheduler': 'true', 'lr_scheduler_step': 'auto', 'lr_scheduler_factor': 'auto', 'lr_scheduler_minimum_lr': 'auto', 'positive_example_weight_mult': '1.0', 'balance_multiclass_weights': 'false', 'normalize_data': 'true', 'normalize_label': 'auto', 'unbias_data': 'auto', 'unbias_label': 'auto', 'num_point_for_scaler': '10000', '_kvstore': 'auto', '_num_gpus': 'auto', '_num_kv_servers': 'auto', '_log_level': 'info', '_tuning_objective_metric': '', 'early_stopping_patience': '3', 'early_stopping_tolerance': '0.001', '_enable_profiler': 'false', 'predictor_type': 'regressor'}
[06/16/2022 09:08:32 WARNING 139637139089216] Loggers have already been setup.
Process 1 is a worker.
[06/16/2022 09:08:32 INFO 139637139089216] Using default worker.
[06/16/2022 09:08:33 INFO 139637139089216] Checkpoint loading and saving are disabled.
[06/16/2022 09:08:33 INFO 139637139089216] Create Store: local
[06/16/2022 09:08:33 INFO 139637139089216] Scaler algorithm parameters
 <algorithm.scaler.ScalerAlgorithmStable object at 0x7eff58958050>
[06/16/2022 09:08:33 INFO 139637139089216] Scaling model computed with parameters:
 {'stdev_label': 
[9.34456]
<NDArray 1 @cpu(0)>, 'stdev_weight': 
[8.76453018e+00 2.35804825e+01 6.91447353e+00 2.53265083e-01
 1.16866864e-01 7.11425126e-01 2.81710987e+01 2.16355181e+00
 8.68270397e+00 1.67831055e+02 2.19965935e+00 7.17631102e+00]
<NDArray 12 @cpu(0)>, 'mean_label': 
[22.730888]
<NDArray 1 @cpu(0)>, 'mean_weight': 
[3.6912031e+00 1.1747775e+01 1.1070757e+01 6.8888903e-02 5.5524135e-01
 6.2997293e+00 6.8247337e+01 3.8249063e+00 9.5533333e+00 4.0815112e+02
 1.8405781e+01 1.2513045e+01]
<NDArray 12 @cpu(0)>}
[06/16/2022 09:08:33 INFO 139637139089216] nvidia-smi: took 0.034 seconds to run.
[06/16/2022 09:08:33 INFO 139637139089216] nvidia-smi identified 0 GPUs.
[06/16/2022 09:08:33 INFO 139637139089216] Number of GPUs being used: 0
#metrics {"StartTime": 1655370513.6247761, "EndTime": 1655370513.6248353, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "Meta": "init_train_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 485.0, "count": 1, "min": 485, "max": 485}, "Total Batches Seen": {"sum": 17.0, "count": 1, "min": 17, "max": 17}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 2.0, "count": 1, "min": 2, "max": 2}, "Number of Records Since Last Reset": {"sum": 0.0, "count": 1, "min": 0, "max": 0}, "Number of Batches Since Last Reset": {"sum": 0.0, "count": 1, "min": 0, "max": 0}}}
#metrics {"StartTime": 1655370514.1997795, "EndTime": 1655370514.199862, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.7342255814870199, "count": 1, "min": 0.7342255814870199, "max": 0.7342255814870199}}}
#metrics {"StartTime": 1655370514.200417, "EndTime": 1655370514.200444, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.8231188530392117, "count": 1, "min": 0.8231188530392117, "max": 0.8231188530392117}}}
#metrics {"StartTime": 1655370514.2007773, "EndTime": 1655370514.2008433, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.8803848605685763, "count": 1, "min": 0.8803848605685763, "max": 0.8803848605685763}}}
#metrics {"StartTime": 1655370514.2011683, "EndTime": 1655370514.201206, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.8761744774712457, "count": 1, "min": 0.8761744774712457, "max": 0.8761744774712457}}}
#metrics {"StartTime": 1655370514.2014973, "EndTime": 1655370514.2015386, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.656442772547404, "count": 1, "min": 0.656442772547404, "max": 0.656442772547404}}}
#metrics {"StartTime": 1655370514.2018526, "EndTime": 1655370514.2018926, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.9078461117214627, "count": 1, "min": 0.9078461117214627, "max": 0.9078461117214627}}}
#metrics {"StartTime": 1655370514.2021906, "EndTime": 1655370514.202229, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.6571193493737115, "count": 1, "min": 0.6571193493737115, "max": 0.6571193493737115}}}
#metrics {"StartTime": 1655370514.2025762, "EndTime": 1655370514.2026186, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.8111394161648221, "count": 1, "min": 0.8111394161648221, "max": 0.8111394161648221}}}
#metrics {"StartTime": 1655370514.2029133, "EndTime": 1655370514.2029514, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.7890813573201497, "count": 1, "min": 0.7890813573201497, "max": 0.7890813573201497}}}
#metrics {"StartTime": 1655370514.2032437, "EndTime": 1655370514.2032828, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.6199219311608208, "count": 1, "min": 0.6199219311608208, "max": 0.6199219311608208}}}
#metrics {"StartTime": 1655370514.2035882, "EndTime": 1655370514.2036273, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.9667545784844293, "count": 1, "min": 0.9667545784844293, "max": 0.9667545784844293}}}
#metrics {"StartTime": 1655370514.203925, "EndTime": 1655370514.2039638, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.8675954903496637, "count": 1, "min": 0.8675954903496637, "max": 0.8675954903496637}}}
#metrics {"StartTime": 1655370514.2042556, "EndTime": 1655370514.204296, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.87180573993259, "count": 1, "min": 0.87180573993259, "max": 0.87180573993259}}}
#metrics {"StartTime": 1655370514.204613, "EndTime": 1655370514.2046492, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.6551772255367703, "count": 1, "min": 0.6551772255367703, "max": 0.6551772255367703}}}
#metrics {"StartTime": 1655370514.204947, "EndTime": 1655370514.2049868, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.8131753158569336, "count": 1, "min": 0.8131753158569336, "max": 0.8131753158569336}}}
#metrics {"StartTime": 1655370514.205287, "EndTime": 1655370514.205329, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.8455197090572781, "count": 1, "min": 0.8455197090572781, "max": 0.8455197090572781}}}
#metrics {"StartTime": 1655370514.205626, "EndTime": 1655370514.2056801, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.7571575673421224, "count": 1, "min": 0.7571575673421224, "max": 0.7571575673421224}}}
#metrics {"StartTime": 1655370514.205997, "EndTime": 1655370514.2060547, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.8838858689202203, "count": 1, "min": 0.8838858689202203, "max": 0.8838858689202203}}}
#metrics {"StartTime": 1655370514.2063773, "EndTime": 1655370514.2064557, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.7011345269944933, "count": 1, "min": 0.7011345269944933, "max": 0.7011345269944933}}}
#metrics {"StartTime": 1655370514.2067807, "EndTime": 1655370514.2068167, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.784123322168986, "count": 1, "min": 0.784123322168986, "max": 0.784123322168986}}}
#metrics {"StartTime": 1655370514.207115, "EndTime": 1655370514.2071867, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.8399419085184733, "count": 1, "min": 0.8399419085184733, "max": 0.8399419085184733}}}
#metrics {"StartTime": 1655370514.2074873, "EndTime": 1655370514.2075248, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.676321226755778, "count": 1, "min": 0.676321226755778, "max": 0.676321226755778}}}
#metrics {"StartTime": 1655370514.2078495, "EndTime": 1655370514.2078874, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.6817984845903184, "count": 1, "min": 0.6817984845903184, "max": 0.6817984845903184}}}
#metrics {"StartTime": 1655370514.208195, "EndTime": 1655370514.208255, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.7986219829983181, "count": 1, "min": 0.7986219829983181, "max": 0.7986219829983181}}}
#metrics {"StartTime": 1655370514.208583, "EndTime": 1655370514.2086253, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9477216720581054, "count": 1, "min": 0.9477216720581054, "max": 0.9477216720581054}}}
#metrics {"StartTime": 1655370514.2089238, "EndTime": 1655370514.2089608, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 1.0295849715338814, "count": 1, "min": 1.0295849715338814, "max": 1.0295849715338814}}}
#metrics {"StartTime": 1655370514.2092505, "EndTime": 1655370514.2092884, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9801304117838542, "count": 1, "min": 0.9801304117838542, "max": 0.9801304117838542}}}
#metrics {"StartTime": 1655370514.2096825, "EndTime": 1655370514.2097063, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 1.0793715371025934, "count": 1, "min": 1.0793715371025934, "max": 1.0793715371025934}}}
#metrics {"StartTime": 1655370514.210022, "EndTime": 1655370514.210062, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 1.3322255198160808, "count": 1, "min": 1.3322255198160808, "max": 1.3322255198160808}}}
#metrics {"StartTime": 1655370514.210389, "EndTime": 1655370514.2104301, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 1.1104522429572212, "count": 1, "min": 1.1104522429572212, "max": 1.1104522429572212}}}
#metrics {"StartTime": 1655370514.2107415, "EndTime": 1655370514.2107806, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 1.1085011821322972, "count": 1, "min": 1.1085011821322972, "max": 1.1085011821322972}}}
#metrics {"StartTime": 1655370514.2110868, "EndTime": 1655370514.2111309, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 1.1517539087931314, "count": 1, "min": 1.1517539087931314, "max": 1.1517539087931314}}}
[06/16/2022 09:08:34 INFO 139637139089216] #quality_metric: host=algo-1, epoch=0, train mse_objective <loss>=0.7342255814870199
#metrics {"StartTime": 1655370514.3148293, "EndTime": 1655370514.3149238, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 27.402461033241423, "count": 1, "min": 27.402461033241423, "max": 27.402461033241423}}}
#metrics {"StartTime": 1655370514.3155208, "EndTime": 1655370514.3155527, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 30.773424335554534, "count": 1, "min": 30.773424335554534, "max": 30.773424335554534}}}
#metrics {"StartTime": 1655370514.315889, "EndTime": 1655370514.3159294, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 38.51581049900429, "count": 1, "min": 38.51581049900429, "max": 38.51581049900429}}}
#metrics {"StartTime": 1655370514.3162339, "EndTime": 1655370514.3162794, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 33.189655378753066, "count": 1, "min": 33.189655378753066, "max": 33.189655378753066}}}
#metrics {"StartTime": 1655370514.3166046, "EndTime": 1655370514.3166413, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 35.38766299977022, "count": 1, "min": 35.38766299977022, "max": 35.38766299977022}}}
#metrics {"StartTime": 1655370514.316944, "EndTime": 1655370514.3170278, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 31.813827215456495, "count": 1, "min": 31.813827215456495, "max": 31.813827215456495}}}
#metrics {"StartTime": 1655370514.3173647, "EndTime": 1655370514.3174055, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 34.474498075597424, "count": 1, "min": 34.474498075597424, "max": 34.474498075597424}}}
#metrics {"StartTime": 1655370514.3177202, "EndTime": 1655370514.3177552, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 39.658028396905635, "count": 1, "min": 39.658028396905635, "max": 39.658028396905635}}}
#metrics {"StartTime": 1655370514.3180368, "EndTime": 1655370514.3180761, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 29.30691588158701, "count": 1, "min": 29.30691588158701, "max": 29.30691588158701}}}
#metrics {"StartTime": 1655370514.3183806, "EndTime": 1655370514.318424, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 22.185959759880514, "count": 1, "min": 22.185959759880514, "max": 22.185959759880514}}}
#metrics {"StartTime": 1655370514.3187418, "EndTime": 1655370514.3187768, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 36.23678409352022, "count": 1, "min": 36.23678409352022, "max": 36.23678409352022}}}
#metrics {"StartTime": 1655370514.319071, "EndTime": 1655370514.31911, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 37.74798105277267, "count": 1, "min": 37.74798105277267, "max": 37.74798105277267}}}
#metrics {"StartTime": 1655370514.3194046, "EndTime": 1655370514.3194447, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 34.08496692133885, "count": 1, "min": 34.08496692133885, "max": 34.08496692133885}}}
#metrics {"StartTime": 1655370514.3198159, "EndTime": 1655370514.3198373, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 34.66747927198223, "count": 1, "min": 34.66747927198223, "max": 34.66747927198223}}}
#metrics {"StartTime": 1655370514.320155, "EndTime": 1655370514.3201954, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 37.22522571040135, "count": 1, "min": 37.22522571040135, "max": 37.22522571040135}}}
#metrics {"StartTime": 1655370514.320503, "EndTime": 1655370514.3205466, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 35.7627383961397, "count": 1, "min": 35.7627383961397, "max": 35.7627383961397}}}
#metrics {"StartTime": 1655370514.3208497, "EndTime": 1655370514.3208885, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 31.76248946844363, "count": 1, "min": 31.76248946844363, "max": 31.76248946844363}}}
#metrics {"StartTime": 1655370514.3212028, "EndTime": 1655370514.3212395, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 33.12425561044731, "count": 1, "min": 33.12425561044731, "max": 33.12425561044731}}}
#metrics {"StartTime": 1655370514.3215368, "EndTime": 1655370514.3215775, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 28.10427916283701, "count": 1, "min": 28.10427916283701, "max": 28.10427916283701}}}
#metrics {"StartTime": 1655370514.3218973, "EndTime": 1655370514.3219354, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 32.98116885914522, "count": 1, "min": 32.98116885914522, "max": 32.98116885914522}}}
#metrics {"StartTime": 1655370514.3222327, "EndTime": 1655370514.3222723, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 26.26614559397978, "count": 1, "min": 26.26614559397978, "max": 26.26614559397978}}}
#metrics {"StartTime": 1655370514.322598, "EndTime": 1655370514.3226388, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 25.331933115042894, "count": 1, "min": 25.331933115042894, "max": 25.331933115042894}}}
#metrics {"StartTime": 1655370514.3229315, "EndTime": 1655370514.3229926, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 25.81375062231924, "count": 1, "min": 25.81375062231924, "max": 25.81375062231924}}}
#metrics {"StartTime": 1655370514.3232965, "EndTime": 1655370514.323337, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 29.230767941942403, "count": 1, "min": 29.230767941942403, "max": 29.230767941942403}}}
#metrics {"StartTime": 1655370514.323641, "EndTime": 1655370514.3236794, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 60.009633980545345, "count": 1, "min": 60.009633980545345, "max": 60.009633980545345}}}
#metrics {"StartTime": 1655370514.323973, "EndTime": 1655370514.3240116, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 62.05478084788603, "count": 1, "min": 62.05478084788603, "max": 62.05478084788603}}}
#metrics {"StartTime": 1655370514.3243523, "EndTime": 1655370514.324393, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 63.59229712392769, "count": 1, "min": 63.59229712392769, "max": 63.59229712392769}}}
#metrics {"StartTime": 1655370514.3247035, "EndTime": 1655370514.3247397, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 58.148657705269606, "count": 1, "min": 58.148657705269606, "max": 58.148657705269606}}}
#metrics {"StartTime": 1655370514.325044, "EndTime": 1655370514.3250837, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 74.61402324601715, "count": 1, "min": 74.61402324601715, "max": 74.61402324601715}}}
#metrics {"StartTime": 1655370514.3253865, "EndTime": 1655370514.3254254, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 83.38874368106617, "count": 1, "min": 83.38874368106617, "max": 83.38874368106617}}}
#metrics {"StartTime": 1655370514.3257341, "EndTime": 1655370514.3257933, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 81.56805060891544, "count": 1, "min": 81.56805060891544, "max": 81.56805060891544}}}
#metrics {"StartTime": 1655370514.3260648, "EndTime": 1655370514.326102, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 98.70984126072304, "count": 1, "min": 98.70984126072304, "max": 98.70984126072304}}}
[06/16/2022 09:08:34 INFO 139637139089216] #quality_metric: host=algo-1, epoch=0, validation mse_objective <loss>=27.402461033241423
[06/16/2022 09:08:34 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=0, criteria=mse_objective, value=22.185959759880514
[06/16/2022 09:08:34 INFO 139637139089216] Epoch 0: Loss improved. Updating best model
[06/16/2022 09:08:34 INFO 139637139089216] Saving model for epoch: 0
[06/16/2022 09:08:34 INFO 139637139089216] Saved checkpoint to "/tmp/tmp7s2wp7d_/mx-mod-0000.params"
[06/16/2022 09:08:34 INFO 139637139089216] #progress_metric: host=algo-1, completed 6.666666666666667 % of epochs
#metrics {"StartTime": 1655370513.6251845, "EndTime": 1655370514.3389657, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 0, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 940.0, "count": 1, "min": 940, "max": 940}, "Total Batches Seen": {"sum": 33.0, "count": 1, "min": 33, "max": 33}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 3.0, "count": 1, "min": 3, "max": 3}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:34 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=637.351961759049 records/second
#metrics {"StartTime": 1655370514.793661, "EndTime": 1655370514.7937362, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.5166595474878947, "count": 1, "min": 0.5166595474878947, "max": 0.5166595474878947}}}
#metrics {"StartTime": 1655370514.7939193, "EndTime": 1655370514.7939372, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.5506460995144314, "count": 1, "min": 0.5506460995144314, "max": 0.5506460995144314}}}
#metrics {"StartTime": 1655370514.7942128, "EndTime": 1655370514.794322, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.6185988553365072, "count": 1, "min": 0.6185988553365072, "max": 0.6185988553365072}}}
#metrics {"StartTime": 1655370514.7944574, "EndTime": 1655370514.7945628, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.6008805370330811, "count": 1, "min": 0.6008805370330811, "max": 0.6008805370330811}}}
#metrics {"StartTime": 1655370514.794682, "EndTime": 1655370514.7947862, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.37833006223042803, "count": 1, "min": 0.37833006223042803, "max": 0.37833006223042803}}}
#metrics {"StartTime": 1655370514.7948918, "EndTime": 1655370514.7949944, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.4203815163506402, "count": 1, "min": 0.4203815163506402, "max": 0.4203815163506402}}}
#metrics {"StartTime": 1655370514.7951074, "EndTime": 1655370514.7951229, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.38922022501627607, "count": 1, "min": 0.38922022501627607, "max": 0.38922022501627607}}}
#metrics {"StartTime": 1655370514.7952247, "EndTime": 1655370514.79524, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.44315323193868, "count": 1, "min": 0.44315323193868, "max": 0.44315323193868}}}
#metrics {"StartTime": 1655370514.7955458, "EndTime": 1655370514.795566, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.5617256683773465, "count": 1, "min": 0.5617256683773465, "max": 0.5617256683773465}}}
#metrics {"StartTime": 1655370514.795666, "EndTime": 1655370514.7956815, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.4656750954522027, "count": 1, "min": 0.4656750954522027, "max": 0.4656750954522027}}}
#metrics {"StartTime": 1655370514.7957923, "EndTime": 1655370514.7959692, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.5940322356753879, "count": 1, "min": 0.5940322356753879, "max": 0.5940322356753879}}}
#metrics {"StartTime": 1655370514.796095, "EndTime": 1655370514.7961931, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.6260889795091417, "count": 1, "min": 0.6260889795091417, "max": 0.6260889795091417}}}
#metrics {"StartTime": 1655370514.796305, "EndTime": 1655370514.7964015, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.4245594278971354, "count": 1, "min": 0.4245594278971354, "max": 0.4245594278971354}}}
#metrics {"StartTime": 1655370514.7965064, "EndTime": 1655370514.7966027, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.3822390651702881, "count": 1, "min": 0.3822390651702881, "max": 0.3822390651702881}}}
#metrics {"StartTime": 1655370514.796765, "EndTime": 1655370514.7968645, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.43491101688808864, "count": 1, "min": 0.43491101688808864, "max": 0.43491101688808864}}}
#metrics {"StartTime": 1655370514.7970545, "EndTime": 1655370514.7971013, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.43132063547770183, "count": 1, "min": 0.43132063547770183, "max": 0.43132063547770183}}}
#metrics {"StartTime": 1655370514.7972143, "EndTime": 1655370514.79732, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.5299906879001194, "count": 1, "min": 0.5299906879001194, "max": 0.5299906879001194}}}
#metrics {"StartTime": 1655370514.7975247, "EndTime": 1655370514.797551, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.5932350529564752, "count": 1, "min": 0.5932350529564752, "max": 0.5932350529564752}}}
#metrics {"StartTime": 1655370514.7977629, "EndTime": 1655370514.797783, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.5233783536487155, "count": 1, "min": 0.5233783536487155, "max": 0.5233783536487155}}}
#metrics {"StartTime": 1655370514.7979352, "EndTime": 1655370514.7979546, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.5680733150906033, "count": 1, "min": 0.5680733150906033, "max": 0.5680733150906033}}}
#metrics {"StartTime": 1655370514.7980409, "EndTime": 1655370514.7980552, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.5522896650102403, "count": 1, "min": 0.5522896650102403, "max": 0.5522896650102403}}}
#metrics {"StartTime": 1655370514.7984867, "EndTime": 1655370514.7985315, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.4333591233359443, "count": 1, "min": 0.4333591233359443, "max": 0.4333591233359443}}}
#metrics {"StartTime": 1655370514.7988224, "EndTime": 1655370514.798846, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4411356327268812, "count": 1, "min": 0.4411356327268812, "max": 0.4411356327268812}}}
#metrics {"StartTime": 1655370514.7991967, "EndTime": 1655370514.7992182, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.5438913355933296, "count": 1, "min": 0.5438913355933296, "max": 0.5438913355933296}}}
#metrics {"StartTime": 1655370514.7995625, "EndTime": 1655370514.7995846, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9584204080369737, "count": 1, "min": 0.9584204080369737, "max": 0.9584204080369737}}}
#metrics {"StartTime": 1655370514.7999318, "EndTime": 1655370514.7999597, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9392160394456651, "count": 1, "min": 0.9392160394456651, "max": 0.9392160394456651}}}
#metrics {"StartTime": 1655370514.8002694, "EndTime": 1655370514.8002915, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9485654428270128, "count": 1, "min": 0.9485654428270128, "max": 0.9485654428270128}}}
#metrics {"StartTime": 1655370514.8005514, "EndTime": 1655370514.8005733, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9190441343519423, "count": 1, "min": 0.9190441343519423, "max": 0.9190441343519423}}}
#metrics {"StartTime": 1655370514.8006892, "EndTime": 1655370514.8007975, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 1.067823766072591, "count": 1, "min": 1.067823766072591, "max": 1.067823766072591}}}
#metrics {"StartTime": 1655370514.8011131, "EndTime": 1655370514.8011549, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 1.0263462956746419, "count": 1, "min": 1.0263462956746419, "max": 1.0263462956746419}}}
#metrics {"StartTime": 1655370514.8014686, "EndTime": 1655370514.8015127, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 1.0256339263916017, "count": 1, "min": 1.0256339263916017, "max": 1.0256339263916017}}}
#metrics {"StartTime": 1655370514.8018544, "EndTime": 1655370514.8019047, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 1.0389580175611708, "count": 1, "min": 1.0389580175611708, "max": 1.0389580175611708}}}
[06/16/2022 09:08:34 INFO 139637139089216] #quality_metric: host=algo-1, epoch=1, train mse_objective <loss>=0.5166595474878947
#metrics {"StartTime": 1655370514.8894382, "EndTime": 1655370514.8895233, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 20.534227558210784, "count": 1, "min": 20.534227558210784, "max": 20.534227558210784}}}
#metrics {"StartTime": 1655370514.8900254, "EndTime": 1655370514.890074, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 21.64132391237745, "count": 1, "min": 21.64132391237745, "max": 21.64132391237745}}}
#metrics {"StartTime": 1655370514.8903944, "EndTime": 1655370514.8904364, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 27.875167547487745, "count": 1, "min": 27.875167547487745, "max": 27.875167547487745}}}
#metrics {"StartTime": 1655370514.8907256, "EndTime": 1655370514.8907456, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 23.328314089307597, "count": 1, "min": 23.328314089307597, "max": 23.328314089307597}}}
#metrics {"StartTime": 1655370514.8910687, "EndTime": 1655370514.89111, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 12.65540687710631, "count": 1, "min": 12.65540687710631, "max": 12.65540687710631}}}
#metrics {"StartTime": 1655370514.891395, "EndTime": 1655370514.8914142, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 23.28372970281863, "count": 1, "min": 23.28372970281863, "max": 23.28372970281863}}}
#metrics {"StartTime": 1655370514.891733, "EndTime": 1655370514.8917713, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 13.133567062078738, "count": 1, "min": 13.133567062078738, "max": 13.133567062078738}}}
#metrics {"StartTime": 1655370514.8920536, "EndTime": 1655370514.8920732, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 27.100325042126226, "count": 1, "min": 27.100325042126226, "max": 27.100325042126226}}}
#metrics {"StartTime": 1655370514.893827, "EndTime": 1655370514.8938773, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 21.418879490272673, "count": 1, "min": 21.418879490272673, "max": 21.418879490272673}}}
#metrics {"StartTime": 1655370514.8942196, "EndTime": 1655370514.8942635, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 17.733844831878063, "count": 1, "min": 17.733844831878063, "max": 17.733844831878063}}}
#metrics {"StartTime": 1655370514.894603, "EndTime": 1655370514.8946478, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 23.06170534620098, "count": 1, "min": 23.06170534620098, "max": 23.06170534620098}}}
#metrics {"StartTime": 1655370514.8950067, "EndTime": 1655370514.8950486, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 28.42096066942402, "count": 1, "min": 28.42096066942402, "max": 28.42096066942402}}}
#metrics {"StartTime": 1655370514.8953843, "EndTime": 1655370514.8954334, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 25.944401161343443, "count": 1, "min": 25.944401161343443, "max": 25.944401161343443}}}
#metrics {"StartTime": 1655370514.8957503, "EndTime": 1655370514.8957975, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 13.409918691597733, "count": 1, "min": 13.409918691597733, "max": 13.409918691597733}}}
#metrics {"StartTime": 1655370514.8961208, "EndTime": 1655370514.8961651, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 26.13724532781863, "count": 1, "min": 26.13724532781863, "max": 26.13724532781863}}}
#metrics {"StartTime": 1655370514.8964906, "EndTime": 1655370514.8965366, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 26.59408509497549, "count": 1, "min": 26.59408509497549, "max": 26.59408509497549}}}
#metrics {"StartTime": 1655370514.8968673, "EndTime": 1655370514.89692, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 24.054742551317403, "count": 1, "min": 24.054742551317403, "max": 24.054742551317403}}}
#metrics {"StartTime": 1655370514.897243, "EndTime": 1655370514.8972852, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 23.887010761335784, "count": 1, "min": 23.887010761335784, "max": 23.887010761335784}}}
#metrics {"StartTime": 1655370514.897665, "EndTime": 1655370514.8976884, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 22.27529967064951, "count": 1, "min": 22.27529967064951, "max": 22.27529967064951}}}
#metrics {"StartTime": 1655370514.8980236, "EndTime": 1655370514.89807, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 24.790613511029413, "count": 1, "min": 24.790613511029413, "max": 24.790613511029413}}}
#metrics {"StartTime": 1655370514.8984141, "EndTime": 1655370514.8984637, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 22.136188581878063, "count": 1, "min": 22.136188581878063, "max": 22.136188581878063}}}
#metrics {"StartTime": 1655370514.8987782, "EndTime": 1655370514.8988209, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 21.865616143918505, "count": 1, "min": 21.865616143918505, "max": 21.865616143918505}}}
#metrics {"StartTime": 1655370514.8991504, "EndTime": 1655370514.8991938, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 21.634711770450366, "count": 1, "min": 21.634711770450366, "max": 21.634711770450366}}}
#metrics {"StartTime": 1655370514.8995178, "EndTime": 1655370514.8995636, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 22.551690793504903, "count": 1, "min": 22.551690793504903, "max": 22.551690793504903}}}
#metrics {"StartTime": 1655370514.899884, "EndTime": 1655370514.8999379, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 63.2322567210478, "count": 1, "min": 63.2322567210478, "max": 63.2322567210478}}}
#metrics {"StartTime": 1655370514.9002714, "EndTime": 1655370514.900316, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 59.91164742264093, "count": 1, "min": 59.91164742264093, "max": 59.91164742264093}}}
#metrics {"StartTime": 1655370514.9006438, "EndTime": 1655370514.9007075, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.350928213082106, "count": 1, "min": 61.350928213082106, "max": 61.350928213082106}}}
#metrics {"StartTime": 1655370514.901036, "EndTime": 1655370514.90108, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 60.662939931832106, "count": 1, "min": 60.662939931832106, "max": 60.662939931832106}}}
#metrics {"StartTime": 1655370514.9013925, "EndTime": 1655370514.901439, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 58.422523648131126, "count": 1, "min": 58.422523648131126, "max": 58.422523648131126}}}
#metrics {"StartTime": 1655370514.9017863, "EndTime": 1655370514.9018378, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 59.99669213388481, "count": 1, "min": 59.99669213388481, "max": 59.99669213388481}}}
#metrics {"StartTime": 1655370514.9021795, "EndTime": 1655370514.9022245, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 59.20013786764706, "count": 1, "min": 59.20013786764706, "max": 59.20013786764706}}}
#metrics {"StartTime": 1655370514.9025545, "EndTime": 1655370514.9026, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 59.66762886795343, "count": 1, "min": 59.66762886795343, "max": 59.66762886795343}}}
[06/16/2022 09:08:34 INFO 139637139089216] #quality_metric: host=algo-1, epoch=1, validation mse_objective <loss>=20.534227558210784
[06/16/2022 09:08:34 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=1, criteria=mse_objective, value=12.65540687710631
[06/16/2022 09:08:34 INFO 139637139089216] Epoch 1: Loss improved. Updating best model
[06/16/2022 09:08:34 INFO 139637139089216] Saving model for epoch: 1
[06/16/2022 09:08:34 INFO 139637139089216] Saved checkpoint to "/tmp/tmppz33zbgv/mx-mod-0000.params"
[06/16/2022 09:08:34 INFO 139637139089216] #progress_metric: host=algo-1, completed 13.333333333333334 % of epochs
#metrics {"StartTime": 1655370514.3393297, "EndTime": 1655370514.9135427, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 1, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 1395.0, "count": 1, "min": 1395, "max": 1395}, "Total Batches Seen": {"sum": 49.0, "count": 1, "min": 49, "max": 49}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 4.0, "count": 1, "min": 4, "max": 4}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:34 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=791.7530590891478 records/second
#metrics {"StartTime": 1655370515.4228156, "EndTime": 1655370515.4228654, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.4330157709121704, "count": 1, "min": 0.4330157709121704, "max": 0.4330157709121704}}}
#metrics {"StartTime": 1655370515.4229321, "EndTime": 1655370515.4229438, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.4542000749376085, "count": 1, "min": 0.4542000749376085, "max": 0.4542000749376085}}}
#metrics {"StartTime": 1655370515.4229865, "EndTime": 1655370515.4229977, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.507329733106825, "count": 1, "min": 0.507329733106825, "max": 0.507329733106825}}}
#metrics {"StartTime": 1655370515.4230287, "EndTime": 1655370515.4230385, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.48992490821414525, "count": 1, "min": 0.48992490821414525, "max": 0.48992490821414525}}}
#metrics {"StartTime": 1655370515.423068, "EndTime": 1655370515.4230778, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.3084509409798516, "count": 1, "min": 0.3084509409798516, "max": 0.3084509409798516}}}
#metrics {"StartTime": 1655370515.4231062, "EndTime": 1655370515.4231162, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.35135615560743544, "count": 1, "min": 0.35135615560743544, "max": 0.35135615560743544}}}
#metrics {"StartTime": 1655370515.4231443, "EndTime": 1655370515.4231539, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.3071141947640313, "count": 1, "min": 0.3071141947640313, "max": 0.3071141947640313}}}
#metrics {"StartTime": 1655370515.423182, "EndTime": 1655370515.4231913, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.39416544066535103, "count": 1, "min": 0.39416544066535103, "max": 0.39416544066535103}}}
#metrics {"StartTime": 1655370515.4232192, "EndTime": 1655370515.4232285, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.4570247713724772, "count": 1, "min": 0.4570247713724772, "max": 0.4570247713724772}}}
#metrics {"StartTime": 1655370515.4232562, "EndTime": 1655370515.4232652, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.3919076877170139, "count": 1, "min": 0.3919076877170139, "max": 0.3919076877170139}}}
#metrics {"StartTime": 1655370515.4232929, "EndTime": 1655370515.4233022, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.4911240275700887, "count": 1, "min": 0.4911240275700887, "max": 0.4911240275700887}}}
#metrics {"StartTime": 1655370515.4233298, "EndTime": 1655370515.4233387, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.5087960380978055, "count": 1, "min": 0.5087960380978055, "max": 0.5087960380978055}}}
#metrics {"StartTime": 1655370515.4233673, "EndTime": 1655370515.423376, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.3690242481231689, "count": 1, "min": 0.3690242481231689, "max": 0.3690242481231689}}}
#metrics {"StartTime": 1655370515.4234037, "EndTime": 1655370515.4234135, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.30694677035013834, "count": 1, "min": 0.30694677035013834, "max": 0.30694677035013834}}}
#metrics {"StartTime": 1655370515.4234416, "EndTime": 1655370515.4234507, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.38157987912495933, "count": 1, "min": 0.38157987912495933, "max": 0.38157987912495933}}}
#metrics {"StartTime": 1655370515.4234781, "EndTime": 1655370515.4234872, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.3797152307298448, "count": 1, "min": 0.3797152307298448, "max": 0.3797152307298448}}}
#metrics {"StartTime": 1655370515.4235144, "EndTime": 1655370515.4235237, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.4626873074637519, "count": 1, "min": 0.4626873074637519, "max": 0.4626873074637519}}}
#metrics {"StartTime": 1655370515.4235513, "EndTime": 1655370515.4235604, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.4949516042073568, "count": 1, "min": 0.4949516042073568, "max": 0.4949516042073568}}}
#metrics {"StartTime": 1655370515.423588, "EndTime": 1655370515.42361, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.4552154101265801, "count": 1, "min": 0.4552154101265801, "max": 0.4552154101265801}}}
#metrics {"StartTime": 1655370515.4236434, "EndTime": 1655370515.4236536, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.47885233985053166, "count": 1, "min": 0.47885233985053166, "max": 0.47885233985053166}}}
#metrics {"StartTime": 1655370515.423682, "EndTime": 1655370515.4236915, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.4555201307932536, "count": 1, "min": 0.4555201307932536, "max": 0.4555201307932536}}}
#metrics {"StartTime": 1655370515.4237192, "EndTime": 1655370515.4237285, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.4135795752207438, "count": 1, "min": 0.4135795752207438, "max": 0.4135795752207438}}}
#metrics {"StartTime": 1655370515.423756, "EndTime": 1655370515.423765, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.40918830977545845, "count": 1, "min": 0.40918830977545845, "max": 0.40918830977545845}}}
#metrics {"StartTime": 1655370515.4237924, "EndTime": 1655370515.4238017, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.4457821973164876, "count": 1, "min": 0.4457821973164876, "max": 0.4457821973164876}}}
#metrics {"StartTime": 1655370515.423829, "EndTime": 1655370515.4238384, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9659426667955187, "count": 1, "min": 0.9659426667955187, "max": 0.9659426667955187}}}
#metrics {"StartTime": 1655370515.4238653, "EndTime": 1655370515.4238749, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9553767310248481, "count": 1, "min": 0.9553767310248481, "max": 0.9553767310248481}}}
#metrics {"StartTime": 1655370515.4239028, "EndTime": 1655370515.423913, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9574976030985515, "count": 1, "min": 0.9574976030985515, "max": 0.9574976030985515}}}
#metrics {"StartTime": 1655370515.4239492, "EndTime": 1655370515.4239597, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9598651504516602, "count": 1, "min": 0.9598651504516602, "max": 0.9598651504516602}}}
#metrics {"StartTime": 1655370515.4239879, "EndTime": 1655370515.4239972, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9971387354532878, "count": 1, "min": 0.9971387354532878, "max": 0.9971387354532878}}}
#metrics {"StartTime": 1655370515.4240248, "EndTime": 1655370515.4240344, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.987955216301812, "count": 1, "min": 0.987955216301812, "max": 0.987955216301812}}}
#metrics {"StartTime": 1655370515.424062, "EndTime": 1655370515.424071, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9868502362569174, "count": 1, "min": 0.9868502362569174, "max": 0.9868502362569174}}}
#metrics {"StartTime": 1655370515.4240985, "EndTime": 1655370515.424108, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9949375194973415, "count": 1, "min": 0.9949375194973415, "max": 0.9949375194973415}}}
[06/16/2022 09:08:35 INFO 139637139089216] #quality_metric: host=algo-1, epoch=2, train mse_objective <loss>=0.4330157709121704
#metrics {"StartTime": 1655370515.5262842, "EndTime": 1655370515.5263817, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 18.56622553806679, "count": 1, "min": 18.56622553806679, "max": 18.56622553806679}}}
#metrics {"StartTime": 1655370515.526723, "EndTime": 1655370515.5267525, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 19.290706858915442, "count": 1, "min": 19.290706858915442, "max": 19.290706858915442}}}
#metrics {"StartTime": 1655370515.5271566, "EndTime": 1655370515.5272026, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 22.47011551202512, "count": 1, "min": 22.47011551202512, "max": 22.47011551202512}}}
#metrics {"StartTime": 1655370515.5275395, "EndTime": 1655370515.5275857, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 19.617024141199447, "count": 1, "min": 19.617024141199447, "max": 19.617024141199447}}}
#metrics {"StartTime": 1655370515.5279338, "EndTime": 1655370515.528078, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 16.28774246515012, "count": 1, "min": 16.28774246515012, "max": 16.28774246515012}}}
#metrics {"StartTime": 1655370515.5284405, "EndTime": 1655370515.529872, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 16.712075027765014, "count": 1, "min": 16.712075027765014, "max": 16.712075027765014}}}
#metrics {"StartTime": 1655370515.5302424, "EndTime": 1655370515.5302901, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 16.65838742723652, "count": 1, "min": 16.65838742723652, "max": 16.65838742723652}}}
#metrics {"StartTime": 1655370515.5306304, "EndTime": 1655370515.5306756, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 17.228813021790746, "count": 1, "min": 17.228813021790746, "max": 17.228813021790746}}}
#metrics {"StartTime": 1655370515.5310178, "EndTime": 1655370515.531063, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 18.618967093673408, "count": 1, "min": 18.618967093673408, "max": 18.618967093673408}}}
#metrics {"StartTime": 1655370515.5314093, "EndTime": 1655370515.5314512, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 16.397164737477024, "count": 1, "min": 16.397164737477024, "max": 16.397164737477024}}}
#metrics {"StartTime": 1655370515.531789, "EndTime": 1655370515.5318391, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 19.562892539828432, "count": 1, "min": 19.562892539828432, "max": 19.562892539828432}}}
#metrics {"StartTime": 1655370515.5321634, "EndTime": 1655370515.5322104, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 23.306177475873163, "count": 1, "min": 23.306177475873163, "max": 23.306177475873163}}}
#metrics {"StartTime": 1655370515.5325294, "EndTime": 1655370515.5325742, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 17.138356526692707, "count": 1, "min": 17.138356526692707, "max": 17.138356526692707}}}
#metrics {"StartTime": 1655370515.5329034, "EndTime": 1655370515.5329773, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 16.546471689261644, "count": 1, "min": 16.546471689261644, "max": 16.546471689261644}}}
#metrics {"StartTime": 1655370515.533344, "EndTime": 1655370515.5333664, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 15.802816353592219, "count": 1, "min": 15.802816353592219, "max": 15.802816353592219}}}
#metrics {"StartTime": 1655370515.533725, "EndTime": 1655370515.53377, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 16.945695465686274, "count": 1, "min": 16.945695465686274, "max": 16.945695465686274}}}
#metrics {"StartTime": 1655370515.5340865, "EndTime": 1655370515.5341291, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 21.12312047621783, "count": 1, "min": 21.12312047621783, "max": 21.12312047621783}}}
#metrics {"StartTime": 1655370515.5344512, "EndTime": 1655370515.534495, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 22.38473331227022, "count": 1, "min": 22.38473331227022, "max": 22.38473331227022}}}
#metrics {"StartTime": 1655370515.5348501, "EndTime": 1655370515.534893, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 20.743672090418197, "count": 1, "min": 20.743672090418197, "max": 20.743672090418197}}}
#metrics {"StartTime": 1655370515.535216, "EndTime": 1655370515.5352612, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 21.93802059397978, "count": 1, "min": 21.93802059397978, "max": 21.93802059397978}}}
#metrics {"StartTime": 1655370515.5355756, "EndTime": 1655370515.535621, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 18.39089786305147, "count": 1, "min": 18.39089786305147, "max": 18.39089786305147}}}
#metrics {"StartTime": 1655370515.535933, "EndTime": 1655370515.535975, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.713614071116726, "count": 1, "min": 19.713614071116726, "max": 19.713614071116726}}}
#metrics {"StartTime": 1655370515.5362978, "EndTime": 1655370515.5363426, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.60989320044424, "count": 1, "min": 19.60989320044424, "max": 19.60989320044424}}}
#metrics {"StartTime": 1655370515.5366876, "EndTime": 1655370515.5367286, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 21.67853441425398, "count": 1, "min": 21.67853441425398, "max": 21.67853441425398}}}
#metrics {"StartTime": 1655370515.537061, "EndTime": 1655370515.5371015, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 60.557566923253674, "count": 1, "min": 60.557566923253674, "max": 60.557566923253674}}}
#metrics {"StartTime": 1655370515.537462, "EndTime": 1655370515.5374856, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 61.31462067248774, "count": 1, "min": 61.31462067248774, "max": 61.31462067248774}}}
#metrics {"StartTime": 1655370515.5388012, "EndTime": 1655370515.5388503, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.02584300321691, "count": 1, "min": 61.02584300321691, "max": 61.02584300321691}}}
#metrics {"StartTime": 1655370515.5391831, "EndTime": 1655370515.5392292, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.95382869944853, "count": 1, "min": 61.95382869944853, "max": 61.95382869944853}}}
#metrics {"StartTime": 1655370515.5395522, "EndTime": 1655370515.5395973, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 60.731210746017155, "count": 1, "min": 60.731210746017155, "max": 60.731210746017155}}}
#metrics {"StartTime": 1655370515.5399215, "EndTime": 1655370515.5399704, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 60.673804189644606, "count": 1, "min": 60.673804189644606, "max": 60.673804189644606}}}
#metrics {"StartTime": 1655370515.5403278, "EndTime": 1655370515.5403867, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 60.38941865808823, "count": 1, "min": 60.38941865808823, "max": 60.38941865808823}}}
#metrics {"StartTime": 1655370515.5407243, "EndTime": 1655370515.540768, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 58.48939424402573, "count": 1, "min": 58.48939424402573, "max": 58.48939424402573}}}
[06/16/2022 09:08:35 INFO 139637139089216] #quality_metric: host=algo-1, epoch=2, validation mse_objective <loss>=18.56622553806679
[06/16/2022 09:08:35 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=2, criteria=mse_objective, value=15.802816353592219
[06/16/2022 09:08:35 INFO 139637139089216] Saving model for epoch: 2
[06/16/2022 09:08:35 INFO 139637139089216] Saved checkpoint to "/tmp/tmp5eh525cs/mx-mod-0000.params"
[06/16/2022 09:08:35 INFO 139637139089216] #progress_metric: host=algo-1, completed 20.0 % of epochs
#metrics {"StartTime": 1655370514.91442, "EndTime": 1655370515.5563972, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 2, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 1850.0, "count": 1, "min": 1850, "max": 1850}, "Total Batches Seen": {"sum": 65.0, "count": 1, "min": 65, "max": 65}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 5.0, "count": 1, "min": 5, "max": 5}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:35 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=708.1011329049273 records/second
#metrics {"StartTime": 1655370516.070785, "EndTime": 1655370516.0708687, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.37687147670321997, "count": 1, "min": 0.37687147670321997, "max": 0.37687147670321997}}}
#metrics {"StartTime": 1655370516.0709586, "EndTime": 1655370516.0709722, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.3899355888366699, "count": 1, "min": 0.3899355888366699, "max": 0.3899355888366699}}}
#metrics {"StartTime": 1655370516.0710146, "EndTime": 1655370516.0710251, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.42631299919552273, "count": 1, "min": 0.42631299919552273, "max": 0.42631299919552273}}}
#metrics {"StartTime": 1655370516.0710595, "EndTime": 1655370516.0710688, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.40993305418226456, "count": 1, "min": 0.40993305418226456, "max": 0.40993305418226456}}}
#metrics {"StartTime": 1655370516.0711024, "EndTime": 1655370516.071112, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.30120240105523005, "count": 1, "min": 0.30120240105523005, "max": 0.30120240105523005}}}
#metrics {"StartTime": 1655370516.0711467, "EndTime": 1655370516.0711563, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.3230732864803738, "count": 1, "min": 0.3230732864803738, "max": 0.3230732864803738}}}
#metrics {"StartTime": 1655370516.07119, "EndTime": 1655370516.0711992, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.3055626720852322, "count": 1, "min": 0.3055626720852322, "max": 0.3055626720852322}}}
#metrics {"StartTime": 1655370516.0712337, "EndTime": 1655370516.071244, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.32883591228061254, "count": 1, "min": 0.32883591228061254, "max": 0.32883591228061254}}}
#metrics {"StartTime": 1655370516.0712812, "EndTime": 1655370516.0712903, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.3894065120485094, "count": 1, "min": 0.3894065120485094, "max": 0.3894065120485094}}}
#metrics {"StartTime": 1655370516.0713246, "EndTime": 1655370516.0713346, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.34771293534172903, "count": 1, "min": 0.34771293534172903, "max": 0.34771293534172903}}}
#metrics {"StartTime": 1655370516.071371, "EndTime": 1655370516.0713813, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.41975070582495794, "count": 1, "min": 0.41975070582495794, "max": 0.41975070582495794}}}
#metrics {"StartTime": 1655370516.0714185, "EndTime": 1655370516.0714288, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.42690609667036267, "count": 1, "min": 0.42690609667036267, "max": 0.42690609667036267}}}
#metrics {"StartTime": 1655370516.071467, "EndTime": 1655370516.0714767, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.3297531509399414, "count": 1, "min": 0.3297531509399414, "max": 0.3297531509399414}}}
#metrics {"StartTime": 1655370516.0715125, "EndTime": 1655370516.0715642, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.3046278773413764, "count": 1, "min": 0.3046278773413764, "max": 0.3046278773413764}}}
#metrics {"StartTime": 1655370516.071608, "EndTime": 1655370516.0716183, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.3233711523479886, "count": 1, "min": 0.3233711523479886, "max": 0.3233711523479886}}}
#metrics {"StartTime": 1655370516.071654, "EndTime": 1655370516.0716634, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.3283268467585246, "count": 1, "min": 0.3283268467585246, "max": 0.3283268467585246}}}
#metrics {"StartTime": 1655370516.0716987, "EndTime": 1655370516.071709, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.42747476153903535, "count": 1, "min": 0.42747476153903535, "max": 0.42747476153903535}}}
#metrics {"StartTime": 1655370516.0717459, "EndTime": 1655370516.0717561, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.44515266471438936, "count": 1, "min": 0.44515266471438936, "max": 0.44515266471438936}}}
#metrics {"StartTime": 1655370516.071793, "EndTime": 1655370516.0718045, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.41816701571146647, "count": 1, "min": 0.41816701571146647, "max": 0.41816701571146647}}}
#metrics {"StartTime": 1655370516.0718422, "EndTime": 1655370516.071852, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.43047955830891926, "count": 1, "min": 0.43047955830891926, "max": 0.43047955830891926}}}
#metrics {"StartTime": 1655370516.071889, "EndTime": 1655370516.0718994, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.4163850498199463, "count": 1, "min": 0.4163850498199463, "max": 0.4163850498199463}}}
#metrics {"StartTime": 1655370516.0719368, "EndTime": 1655370516.0719469, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.40793675952487524, "count": 1, "min": 0.40793675952487524, "max": 0.40793675952487524}}}
#metrics {"StartTime": 1655370516.0719829, "EndTime": 1655370516.0719924, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.40354964680141875, "count": 1, "min": 0.40354964680141875, "max": 0.40354964680141875}}}
#metrics {"StartTime": 1655370516.0720303, "EndTime": 1655370516.072041, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.40695947647094727, "count": 1, "min": 0.40695947647094727, "max": 0.40695947647094727}}}
#metrics {"StartTime": 1655370516.0720763, "EndTime": 1655370516.0720863, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9495429229736329, "count": 1, "min": 0.9495429229736329, "max": 0.9495429229736329}}}
#metrics {"StartTime": 1655370516.0721233, "EndTime": 1655370516.0721338, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9583488655090332, "count": 1, "min": 0.9583488655090332, "max": 0.9583488655090332}}}
#metrics {"StartTime": 1655370516.07217, "EndTime": 1655370516.0721803, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9557569122314453, "count": 1, "min": 0.9557569122314453, "max": 0.9557569122314453}}}
#metrics {"StartTime": 1655370516.0722191, "EndTime": 1655370516.0722296, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9582390340169271, "count": 1, "min": 0.9582390340169271, "max": 0.9582390340169271}}}
#metrics {"StartTime": 1655370516.0722644, "EndTime": 1655370516.0722744, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9759493743048774, "count": 1, "min": 0.9759493743048774, "max": 0.9759493743048774}}}
#metrics {"StartTime": 1655370516.072311, "EndTime": 1655370516.0723207, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9639809778001573, "count": 1, "min": 0.9639809778001573, "max": 0.9639809778001573}}}
#metrics {"StartTime": 1655370516.0723572, "EndTime": 1655370516.0723674, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9631755362616645, "count": 1, "min": 0.9631755362616645, "max": 0.9631755362616645}}}
#metrics {"StartTime": 1655370516.072404, "EndTime": 1655370516.0724137, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9622686470879449, "count": 1, "min": 0.9622686470879449, "max": 0.9622686470879449}}}
[06/16/2022 09:08:36 INFO 139637139089216] #quality_metric: host=algo-1, epoch=3, train mse_objective <loss>=0.37687147670321997
#metrics {"StartTime": 1655370516.172835, "EndTime": 1655370516.1729176, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 16.9971098058364, "count": 1, "min": 16.9971098058364, "max": 16.9971098058364}}}
#metrics {"StartTime": 1655370516.1747851, "EndTime": 1655370516.1748104, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 17.806202009612438, "count": 1, "min": 17.806202009612438, "max": 17.806202009612438}}}
#metrics {"StartTime": 1655370516.174856, "EndTime": 1655370516.1748667, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 19.166063495710784, "count": 1, "min": 19.166063495710784, "max": 19.166063495710784}}}
#metrics {"StartTime": 1655370516.1749024, "EndTime": 1655370516.1749117, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 17.397721832873774, "count": 1, "min": 17.397721832873774, "max": 17.397721832873774}}}
#metrics {"StartTime": 1655370516.1749458, "EndTime": 1655370516.174956, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.51783722522212, "count": 1, "min": 14.51783722522212, "max": 14.51783722522212}}}
#metrics {"StartTime": 1655370516.1749904, "EndTime": 1655370516.1749997, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 16.57377415077359, "count": 1, "min": 16.57377415077359, "max": 16.57377415077359}}}
#metrics {"StartTime": 1655370516.1753297, "EndTime": 1655370516.1753535, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.622593898399204, "count": 1, "min": 14.622593898399204, "max": 14.622593898399204}}}
#metrics {"StartTime": 1655370516.1754174, "EndTime": 1655370516.175431, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 15.245516907935048, "count": 1, "min": 15.245516907935048, "max": 15.245516907935048}}}
#metrics {"StartTime": 1655370516.1754696, "EndTime": 1655370516.17548, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 16.654658897250307, "count": 1, "min": 16.654658897250307, "max": 16.654658897250307}}}
#metrics {"StartTime": 1655370516.1755176, "EndTime": 1655370516.1755273, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 14.970694149241728, "count": 1, "min": 14.970694149241728, "max": 14.970694149241728}}}
#metrics {"StartTime": 1655370516.175564, "EndTime": 1655370516.1755745, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 17.889209223728553, "count": 1, "min": 17.889209223728553, "max": 17.889209223728553}}}
#metrics {"StartTime": 1655370516.175613, "EndTime": 1655370516.175624, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 19.786487654143688, "count": 1, "min": 19.786487654143688, "max": 19.786487654143688}}}
#metrics {"StartTime": 1655370516.17566, "EndTime": 1655370516.17567, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 16.704469568589154, "count": 1, "min": 16.704469568589154, "max": 16.704469568589154}}}
#metrics {"StartTime": 1655370516.1757064, "EndTime": 1655370516.1757169, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 14.202775543811274, "count": 1, "min": 14.202775543811274, "max": 14.202775543811274}}}
#metrics {"StartTime": 1655370516.1757562, "EndTime": 1655370516.175766, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 15.529443479051777, "count": 1, "min": 15.529443479051777, "max": 15.529443479051777}}}
#metrics {"StartTime": 1655370516.1758022, "EndTime": 1655370516.1758118, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 15.707206576478248, "count": 1, "min": 15.707206576478248, "max": 15.707206576478248}}}
#metrics {"StartTime": 1655370516.1758478, "EndTime": 1655370516.1758585, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 19.462962431066178, "count": 1, "min": 19.462962431066178, "max": 19.462962431066178}}}
#metrics {"StartTime": 1655370516.1758938, "EndTime": 1655370516.1759043, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 20.904079661649817, "count": 1, "min": 20.904079661649817, "max": 20.904079661649817}}}
#metrics {"StartTime": 1655370516.1759408, "EndTime": 1655370516.1759512, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 18.759546616498163, "count": 1, "min": 18.759546616498163, "max": 18.759546616498163}}}
#metrics {"StartTime": 1655370516.1759884, "EndTime": 1655370516.1759996, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 19.47744571461397, "count": 1, "min": 19.47744571461397, "max": 19.47744571461397}}}
#metrics {"StartTime": 1655370516.1760361, "EndTime": 1655370516.1760464, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 20.019901649624693, "count": 1, "min": 20.019901649624693, "max": 20.019901649624693}}}
#metrics {"StartTime": 1655370516.176085, "EndTime": 1655370516.176095, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.474550134995404, "count": 1, "min": 19.474550134995404, "max": 19.474550134995404}}}
#metrics {"StartTime": 1655370516.1761308, "EndTime": 1655370516.1761403, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.6024169921875, "count": 1, "min": 19.6024169921875, "max": 19.6024169921875}}}
#metrics {"StartTime": 1655370516.1761775, "EndTime": 1655370516.1761882, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 20.623710482728246, "count": 1, "min": 20.623710482728246, "max": 20.623710482728246}}}
#metrics {"StartTime": 1655370516.176223, "EndTime": 1655370516.176233, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.43261479396446, "count": 1, "min": 61.43261479396446, "max": 61.43261479396446}}}
#metrics {"StartTime": 1655370516.1762698, "EndTime": 1655370516.17628, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 61.60102874157476, "count": 1, "min": 61.60102874157476, "max": 61.60102874157476}}}
#metrics {"StartTime": 1655370516.176317, "EndTime": 1655370516.1763268, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.91429227941177, "count": 1, "min": 61.91429227941177, "max": 61.91429227941177}}}
#metrics {"StartTime": 1655370516.1764174, "EndTime": 1655370516.1764324, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.42448634727329, "count": 1, "min": 61.42448634727329, "max": 61.42448634727329}}}
#metrics {"StartTime": 1655370516.1764727, "EndTime": 1655370516.1764836, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 60.93298579197304, "count": 1, "min": 60.93298579197304, "max": 60.93298579197304}}}
#metrics {"StartTime": 1655370516.1765196, "EndTime": 1655370516.1765292, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 62.669333065257355, "count": 1, "min": 62.669333065257355, "max": 62.669333065257355}}}
#metrics {"StartTime": 1655370516.1765668, "EndTime": 1655370516.176577, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 62.98301786534927, "count": 1, "min": 62.98301786534927, "max": 62.98301786534927}}}
#metrics {"StartTime": 1655370516.176643, "EndTime": 1655370516.176657, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 63.35635196461397, "count": 1, "min": 63.35635196461397, "max": 63.35635196461397}}}
[06/16/2022 09:08:36 INFO 139637139089216] #quality_metric: host=algo-1, epoch=3, validation mse_objective <loss>=16.9971098058364
[06/16/2022 09:08:36 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=3, criteria=mse_objective, value=14.202775543811274
[06/16/2022 09:08:36 INFO 139637139089216] Saving model for epoch: 3
[06/16/2022 09:08:36 INFO 139637139089216] Saved checkpoint to "/tmp/tmpt4vx3hms/mx-mod-0000.params"
[06/16/2022 09:08:36 INFO 139637139089216] #progress_metric: host=algo-1, completed 26.666666666666668 % of epochs
#metrics {"StartTime": 1655370515.5573475, "EndTime": 1655370516.1865118, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 3, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 2305.0, "count": 1, "min": 2305, "max": 2305}, "Total Batches Seen": {"sum": 81.0, "count": 1, "min": 81, "max": 81}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 6.0, "count": 1, "min": 6, "max": 6}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:36 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=723.0333929796658 records/second
#metrics {"StartTime": 1655370516.870697, "EndTime": 1655370516.870818, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.3436708143022325, "count": 1, "min": 0.3436708143022325, "max": 0.3436708143022325}}}
#metrics {"StartTime": 1655370516.8709064, "EndTime": 1655370516.8709197, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.35319281154208715, "count": 1, "min": 0.35319281154208715, "max": 0.35319281154208715}}}
#metrics {"StartTime": 1655370516.8709624, "EndTime": 1655370516.8709724, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.3756487560272217, "count": 1, "min": 0.3756487560272217, "max": 0.3756487560272217}}}
#metrics {"StartTime": 1655370516.871007, "EndTime": 1655370516.8710158, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.3615579546822442, "count": 1, "min": 0.3615579546822442, "max": 0.3615579546822442}}}
#metrics {"StartTime": 1655370516.8710494, "EndTime": 1655370516.8710582, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.29766649776034887, "count": 1, "min": 0.29766649776034887, "max": 0.29766649776034887}}}
#metrics {"StartTime": 1655370516.871093, "EndTime": 1655370516.8711028, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.31679604689280194, "count": 1, "min": 0.31679604689280194, "max": 0.31679604689280194}}}
#metrics {"StartTime": 1655370516.8711355, "EndTime": 1655370516.871145, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.2943812190161811, "count": 1, "min": 0.2943812190161811, "max": 0.2943812190161811}}}
#metrics {"StartTime": 1655370516.8711789, "EndTime": 1655370516.8711882, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.3115263631608751, "count": 1, "min": 0.3115263631608751, "max": 0.3115263631608751}}}
#metrics {"StartTime": 1655370516.8712254, "EndTime": 1655370516.8712356, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.3494891103108724, "count": 1, "min": 0.3494891103108724, "max": 0.3494891103108724}}}
#metrics {"StartTime": 1655370516.87127, "EndTime": 1655370516.87128, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.3216774410671658, "count": 1, "min": 0.3216774410671658, "max": 0.3216774410671658}}}
#metrics {"StartTime": 1655370516.8713155, "EndTime": 1655370516.8713257, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.37353869279225665, "count": 1, "min": 0.37353869279225665, "max": 0.37353869279225665}}}
#metrics {"StartTime": 1655370516.8713634, "EndTime": 1655370516.8713737, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.374813134405348, "count": 1, "min": 0.374813134405348, "max": 0.374813134405348}}}
#metrics {"StartTime": 1655370516.87141, "EndTime": 1655370516.8714206, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.3207968584696452, "count": 1, "min": 0.3207968584696452, "max": 0.3207968584696452}}}
#metrics {"StartTime": 1655370516.871459, "EndTime": 1655370516.8714685, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.2979209719763862, "count": 1, "min": 0.2979209719763862, "max": 0.2979209719763862}}}
#metrics {"StartTime": 1655370516.8715043, "EndTime": 1655370516.8715136, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.30769889089796276, "count": 1, "min": 0.30769889089796276, "max": 0.30769889089796276}}}
#metrics {"StartTime": 1655370516.8715494, "EndTime": 1655370516.8715594, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.3148876725302802, "count": 1, "min": 0.3148876725302802, "max": 0.3148876725302802}}}
#metrics {"StartTime": 1655370516.8715959, "EndTime": 1655370516.8716052, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.40367749320136176, "count": 1, "min": 0.40367749320136176, "max": 0.40367749320136176}}}
#metrics {"StartTime": 1655370516.8716412, "EndTime": 1655370516.871651, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.41464082347022163, "count": 1, "min": 0.41464082347022163, "max": 0.41464082347022163}}}
#metrics {"StartTime": 1655370516.871686, "EndTime": 1655370516.8716955, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.39581135167015924, "count": 1, "min": 0.39581135167015924, "max": 0.39581135167015924}}}
#metrics {"StartTime": 1655370516.871733, "EndTime": 1655370516.8717432, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.4019908942116631, "count": 1, "min": 0.4019908942116631, "max": 0.4019908942116631}}}
#metrics {"StartTime": 1655370516.8717804, "EndTime": 1655370516.871791, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.4012196848127577, "count": 1, "min": 0.4012196848127577, "max": 0.4012196848127577}}}
#metrics {"StartTime": 1655370516.8718286, "EndTime": 1655370516.871838, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.40464004516601565, "count": 1, "min": 0.40464004516601565, "max": 0.40464004516601565}}}
#metrics {"StartTime": 1655370516.8718755, "EndTime": 1655370516.8718853, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4007937579684787, "count": 1, "min": 0.4007937579684787, "max": 0.4007937579684787}}}
#metrics {"StartTime": 1655370516.8719227, "EndTime": 1655370516.8719325, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.40211011038886174, "count": 1, "min": 0.40211011038886174, "max": 0.40211011038886174}}}
#metrics {"StartTime": 1655370516.8719683, "EndTime": 1655370516.8719783, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9575183508131239, "count": 1, "min": 0.9575183508131239, "max": 0.9575183508131239}}}
#metrics {"StartTime": 1655370516.872014, "EndTime": 1655370516.8720233, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9550820011562772, "count": 1, "min": 0.9550820011562772, "max": 0.9550820011562772}}}
#metrics {"StartTime": 1655370516.8720608, "EndTime": 1655370516.872071, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9555004607306586, "count": 1, "min": 0.9555004607306586, "max": 0.9555004607306586}}}
#metrics {"StartTime": 1655370516.8721063, "EndTime": 1655370516.8721159, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9545310147603353, "count": 1, "min": 0.9545310147603353, "max": 0.9545310147603353}}}
#metrics {"StartTime": 1655370516.872153, "EndTime": 1655370516.872163, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9699274910820855, "count": 1, "min": 0.9699274910820855, "max": 0.9699274910820855}}}
#metrics {"StartTime": 1655370516.8722003, "EndTime": 1655370516.8722107, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.974634263780382, "count": 1, "min": 0.974634263780382, "max": 0.974634263780382}}}
#metrics {"StartTime": 1655370516.872249, "EndTime": 1655370516.8722591, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9733005502488878, "count": 1, "min": 0.9733005502488878, "max": 0.9733005502488878}}}
#metrics {"StartTime": 1655370516.872295, "EndTime": 1655370516.8723044, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9736014281378852, "count": 1, "min": 0.9736014281378852, "max": 0.9736014281378852}}}
[06/16/2022 09:08:36 INFO 139637139089216] #quality_metric: host=algo-1, epoch=4, train mse_objective <loss>=0.3436708143022325
#metrics {"StartTime": 1655370516.9881036, "EndTime": 1655370516.9881916, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 15.574556238511029, "count": 1, "min": 15.574556238511029, "max": 15.574556238511029}}}
#metrics {"StartTime": 1655370516.9883027, "EndTime": 1655370516.988318, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 16.30394011852788, "count": 1, "min": 16.30394011852788, "max": 16.30394011852788}}}
#metrics {"StartTime": 1655370516.9883606, "EndTime": 1655370516.988371, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 16.47208958046109, "count": 1, "min": 16.47208958046109, "max": 16.47208958046109}}}
#metrics {"StartTime": 1655370516.988405, "EndTime": 1655370516.9884143, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 15.364203957950368, "count": 1, "min": 15.364203957950368, "max": 15.364203957950368}}}
#metrics {"StartTime": 1655370516.9884467, "EndTime": 1655370516.9884555, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.79813699161305, "count": 1, "min": 14.79813699161305, "max": 14.79813699161305}}}
#metrics {"StartTime": 1655370516.988487, "EndTime": 1655370516.9884968, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.44422763001685, "count": 1, "min": 14.44422763001685, "max": 14.44422763001685}}}
#metrics {"StartTime": 1655370516.9885292, "EndTime": 1655370516.9885387, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.71527219286152, "count": 1, "min": 14.71527219286152, "max": 14.71527219286152}}}
#metrics {"StartTime": 1655370516.9885724, "EndTime": 1655370516.9885807, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.899299172794118, "count": 1, "min": 14.899299172794118, "max": 14.899299172794118}}}
#metrics {"StartTime": 1655370516.9886107, "EndTime": 1655370516.9886198, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 15.006472718481923, "count": 1, "min": 15.006472718481923, "max": 15.006472718481923}}}
#metrics {"StartTime": 1655370516.9886525, "EndTime": 1655370516.9886618, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 14.139413871017156, "count": 1, "min": 14.139413871017156, "max": 14.139413871017156}}}
#metrics {"StartTime": 1655370516.988696, "EndTime": 1655370516.9887064, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 16.40328919653799, "count": 1, "min": 16.40328919653799, "max": 16.40328919653799}}}
#metrics {"StartTime": 1655370516.9887393, "EndTime": 1655370516.9887486, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 16.932415532130822, "count": 1, "min": 16.932415532130822, "max": 16.932415532130822}}}
#metrics {"StartTime": 1655370516.9887826, "EndTime": 1655370516.9887927, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.679675532322303, "count": 1, "min": 14.679675532322303, "max": 14.679675532322303}}}
#metrics {"StartTime": 1655370516.9888282, "EndTime": 1655370516.9888382, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 15.153298172296262, "count": 1, "min": 15.153298172296262, "max": 15.153298172296262}}}
#metrics {"StartTime": 1655370516.988875, "EndTime": 1655370516.988885, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.397280225566789, "count": 1, "min": 14.397280225566789, "max": 14.397280225566789}}}
#metrics {"StartTime": 1655370516.9901557, "EndTime": 1655370516.990182, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.570991665709252, "count": 1, "min": 14.570991665709252, "max": 14.570991665709252}}}
#metrics {"StartTime": 1655370516.9902332, "EndTime": 1655370516.9902456, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 18.080331241383274, "count": 1, "min": 18.080331241383274, "max": 18.080331241383274}}}
#metrics {"StartTime": 1655370516.990286, "EndTime": 1655370516.9902983, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 19.017707375919116, "count": 1, "min": 19.017707375919116, "max": 19.017707375919116}}}
#metrics {"StartTime": 1655370516.990343, "EndTime": 1655370516.9903536, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.86836990655637, "count": 1, "min": 17.86836990655637, "max": 17.86836990655637}}}
#metrics {"StartTime": 1655370516.9903905, "EndTime": 1655370516.9904017, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 18.174327177159928, "count": 1, "min": 18.174327177159928, "max": 18.174327177159928}}}
#metrics {"StartTime": 1655370516.9904926, "EndTime": 1655370516.9905066, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.918998568665746, "count": 1, "min": 19.918998568665746, "max": 19.918998568665746}}}
#metrics {"StartTime": 1655370516.990547, "EndTime": 1655370516.9905572, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.432778152765014, "count": 1, "min": 19.432778152765014, "max": 19.432778152765014}}}
#metrics {"StartTime": 1655370516.9905963, "EndTime": 1655370516.990606, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.30176618987439, "count": 1, "min": 19.30176618987439, "max": 19.30176618987439}}}
#metrics {"StartTime": 1655370516.990642, "EndTime": 1655370516.9906526, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.07663681927849, "count": 1, "min": 19.07663681927849, "max": 19.07663681927849}}}
#metrics {"StartTime": 1655370516.9906898, "EndTime": 1655370516.9907, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.44997750076593, "count": 1, "min": 61.44997750076593, "max": 61.44997750076593}}}
#metrics {"StartTime": 1655370516.990737, "EndTime": 1655370516.9907477, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 61.01623295802696, "count": 1, "min": 61.01623295802696, "max": 61.01623295802696}}}
#metrics {"StartTime": 1655370516.9907827, "EndTime": 1655370516.990793, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.438835592830884, "count": 1, "min": 61.438835592830884, "max": 61.438835592830884}}}
#metrics {"StartTime": 1655370516.9908295, "EndTime": 1655370516.9908397, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.31155934053309, "count": 1, "min": 61.31155934053309, "max": 61.31155934053309}}}
#metrics {"StartTime": 1655370516.9908764, "EndTime": 1655370516.9908867, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 60.904258578431374, "count": 1, "min": 60.904258578431374, "max": 60.904258578431374}}}
#metrics {"StartTime": 1655370516.9909248, "EndTime": 1655370516.9909344, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 62.389016544117645, "count": 1, "min": 62.389016544117645, "max": 62.389016544117645}}}
#metrics {"StartTime": 1655370516.9909713, "EndTime": 1655370516.9909818, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 61.984700520833336, "count": 1, "min": 61.984700520833336, "max": 61.984700520833336}}}
#metrics {"StartTime": 1655370516.991019, "EndTime": 1655370516.9910293, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.03021599264706, "count": 1, "min": 62.03021599264706, "max": 62.03021599264706}}}
[06/16/2022 09:08:36 INFO 139637139089216] #quality_metric: host=algo-1, epoch=4, validation mse_objective <loss>=15.574556238511029
[06/16/2022 09:08:36 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=4, criteria=mse_objective, value=14.139413871017156
[06/16/2022 09:08:37 INFO 139637139089216] Saving model for epoch: 4
[06/16/2022 09:08:37 INFO 139637139089216] Saved checkpoint to "/tmp/tmphcewkhmu/mx-mod-0000.params"
[06/16/2022 09:08:37 INFO 139637139089216] #progress_metric: host=algo-1, completed 33.333333333333336 % of epochs
#metrics {"StartTime": 1655370516.186767, "EndTime": 1655370517.0047808, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 4, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 2760.0, "count": 1, "min": 2760, "max": 2760}, "Total Batches Seen": {"sum": 97.0, "count": 1, "min": 97, "max": 97}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 7.0, "count": 1, "min": 7, "max": 7}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:37 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=556.1361122921456 records/second
#metrics {"StartTime": 1655370517.7257173, "EndTime": 1655370517.7257955, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.32255253050062394, "count": 1, "min": 0.32255253050062394, "max": 0.32255253050062394}}}
#metrics {"StartTime": 1655370517.7258809, "EndTime": 1655370517.7258937, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.3298908233642578, "count": 1, "min": 0.3298908233642578, "max": 0.3298908233642578}}}
#metrics {"StartTime": 1655370517.725934, "EndTime": 1655370517.7259436, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.3429497644636366, "count": 1, "min": 0.3429497644636366, "max": 0.3429497644636366}}}
#metrics {"StartTime": 1655370517.7259765, "EndTime": 1655370517.7259853, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.3313139481014676, "count": 1, "min": 0.3313139481014676, "max": 0.3313139481014676}}}
#metrics {"StartTime": 1655370517.7260177, "EndTime": 1655370517.7260263, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.29397834565904407, "count": 1, "min": 0.29397834565904407, "max": 0.29397834565904407}}}
#metrics {"StartTime": 1655370517.7260585, "EndTime": 1655370517.7260673, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.30436518563164605, "count": 1, "min": 0.30436518563164605, "max": 0.30436518563164605}}}
#metrics {"StartTime": 1655370517.726102, "EndTime": 1655370517.7261112, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.293362209531996, "count": 1, "min": 0.293362209531996, "max": 0.293362209531996}}}
#metrics {"StartTime": 1655370517.7261436, "EndTime": 1655370517.726152, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.30295398394266765, "count": 1, "min": 0.30295398394266765, "max": 0.30295398394266765}}}
#metrics {"StartTime": 1655370517.7261856, "EndTime": 1655370517.7261937, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.32513953526814776, "count": 1, "min": 0.32513953526814776, "max": 0.32513953526814776}}}
#metrics {"StartTime": 1655370517.7262278, "EndTime": 1655370517.726238, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.30606098228030737, "count": 1, "min": 0.30606098228030737, "max": 0.30606098228030737}}}
#metrics {"StartTime": 1655370517.7262726, "EndTime": 1655370517.7262812, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.3439723650614421, "count": 1, "min": 0.3439723650614421, "max": 0.3439723650614421}}}
#metrics {"StartTime": 1655370517.726329, "EndTime": 1655370517.72634, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.3409457641177707, "count": 1, "min": 0.3409457641177707, "max": 0.3409457641177707}}}
#metrics {"StartTime": 1655370517.726377, "EndTime": 1655370517.7263873, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.30998548136817083, "count": 1, "min": 0.30998548136817083, "max": 0.30998548136817083}}}
#metrics {"StartTime": 1655370517.7264242, "EndTime": 1655370517.7264345, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.29684680302937827, "count": 1, "min": 0.29684680302937827, "max": 0.29684680302937827}}}
#metrics {"StartTime": 1655370517.7264724, "EndTime": 1655370517.726484, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.30128766112857397, "count": 1, "min": 0.30128766112857397, "max": 0.30128766112857397}}}
#metrics {"StartTime": 1655370517.7265213, "EndTime": 1655370517.726531, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.3036911408106486, "count": 1, "min": 0.3036911408106486, "max": 0.3036911408106486}}}
#metrics {"StartTime": 1655370517.7265656, "EndTime": 1655370517.726575, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.39042553689744736, "count": 1, "min": 0.39042553689744736, "max": 0.39042553689744736}}}
#metrics {"StartTime": 1655370517.726609, "EndTime": 1655370517.7266188, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.3969521771536933, "count": 1, "min": 0.3969521771536933, "max": 0.3969521771536933}}}
#metrics {"StartTime": 1655370517.726655, "EndTime": 1655370517.7266638, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.386775123808119, "count": 1, "min": 0.386775123808119, "max": 0.386775123808119}}}
#metrics {"StartTime": 1655370517.7267008, "EndTime": 1655370517.7267108, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.38949665440453424, "count": 1, "min": 0.38949665440453424, "max": 0.38949665440453424}}}
#metrics {"StartTime": 1655370517.7267456, "EndTime": 1655370517.7267559, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.401328739590115, "count": 1, "min": 0.401328739590115, "max": 0.401328739590115}}}
#metrics {"StartTime": 1655370517.7267923, "EndTime": 1655370517.7268028, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.4058595053354899, "count": 1, "min": 0.4058595053354899, "max": 0.4058595053354899}}}
#metrics {"StartTime": 1655370517.7268393, "EndTime": 1655370517.7268496, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4034149922264947, "count": 1, "min": 0.4034149922264947, "max": 0.4034149922264947}}}
#metrics {"StartTime": 1655370517.7268867, "EndTime": 1655370517.7268965, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.4018051126268175, "count": 1, "min": 0.4018051126268175, "max": 0.4018051126268175}}}
#metrics {"StartTime": 1655370517.7269359, "EndTime": 1655370517.7269452, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9557787068684895, "count": 1, "min": 0.9557787068684895, "max": 0.9557787068684895}}}
#metrics {"StartTime": 1655370517.7269807, "EndTime": 1655370517.7269895, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9561210653516982, "count": 1, "min": 0.9561210653516982, "max": 0.9561210653516982}}}
#metrics {"StartTime": 1655370517.727027, "EndTime": 1655370517.7270365, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9552011320326064, "count": 1, "min": 0.9552011320326064, "max": 0.9552011320326064}}}
#metrics {"StartTime": 1655370517.727072, "EndTime": 1655370517.7270818, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.956407413482666, "count": 1, "min": 0.956407413482666, "max": 0.956407413482666}}}
#metrics {"StartTime": 1655370517.727117, "EndTime": 1655370517.7271266, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9677726003858779, "count": 1, "min": 0.9677726003858779, "max": 0.9677726003858779}}}
#metrics {"StartTime": 1655370517.7271628, "EndTime": 1655370517.7271729, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9717923842536078, "count": 1, "min": 0.9717923842536078, "max": 0.9717923842536078}}}
#metrics {"StartTime": 1655370517.7272089, "EndTime": 1655370517.7272189, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.970803885989719, "count": 1, "min": 0.970803885989719, "max": 0.970803885989719}}}
#metrics {"StartTime": 1655370517.7272556, "EndTime": 1655370517.7272658, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9719388135274252, "count": 1, "min": 0.9719388135274252, "max": 0.9719388135274252}}}
[06/16/2022 09:08:37 INFO 139637139089216] #quality_metric: host=algo-1, epoch=5, train mse_objective <loss>=0.32255253050062394
#metrics {"StartTime": 1655370517.8094492, "EndTime": 1655370517.809523, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 14.737356148514094, "count": 1, "min": 14.737356148514094, "max": 14.737356148514094}}}
#metrics {"StartTime": 1655370517.809606, "EndTime": 1655370517.809619, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 15.317010019339767, "count": 1, "min": 15.317010019339767, "max": 15.317010019339767}}}
#metrics {"StartTime": 1655370517.8096802, "EndTime": 1655370517.8096912, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 14.96641690123315, "count": 1, "min": 14.96641690123315, "max": 14.96641690123315}}}
#metrics {"StartTime": 1655370517.8097253, "EndTime": 1655370517.809734, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 14.3065933526731, "count": 1, "min": 14.3065933526731, "max": 14.3065933526731}}}
#metrics {"StartTime": 1655370517.8097677, "EndTime": 1655370517.8097758, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.402994791666666, "count": 1, "min": 14.402994791666666, "max": 14.402994791666666}}}
#metrics {"StartTime": 1655370517.8098073, "EndTime": 1655370517.8098173, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.821093390969668, "count": 1, "min": 14.821093390969668, "max": 14.821093390969668}}}
#metrics {"StartTime": 1655370517.809851, "EndTime": 1655370517.809861, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.509930778952207, "count": 1, "min": 14.509930778952207, "max": 14.509930778952207}}}
#metrics {"StartTime": 1655370517.8098931, "EndTime": 1655370517.8099024, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.395231359145221, "count": 1, "min": 14.395231359145221, "max": 14.395231359145221}}}
#metrics {"StartTime": 1655370517.8099346, "EndTime": 1655370517.8099427, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 14.16729856004902, "count": 1, "min": 14.16729856004902, "max": 14.16729856004902}}}
#metrics {"StartTime": 1655370517.8099773, "EndTime": 1655370517.8099875, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.732995725145527, "count": 1, "min": 13.732995725145527, "max": 13.732995725145527}}}
#metrics {"StartTime": 1655370517.8100216, "EndTime": 1655370517.8100312, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 15.281980028339461, "count": 1, "min": 15.281980028339461, "max": 15.281980028339461}}}
#metrics {"StartTime": 1655370517.8100672, "EndTime": 1655370517.8100772, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 15.284403483072916, "count": 1, "min": 15.284403483072916, "max": 15.284403483072916}}}
#metrics {"StartTime": 1655370517.810113, "EndTime": 1655370517.8101227, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.81549072265625, "count": 1, "min": 14.81549072265625, "max": 14.81549072265625}}}
#metrics {"StartTime": 1655370517.81016, "EndTime": 1655370517.8101711, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 14.255533854166666, "count": 1, "min": 14.255533854166666, "max": 14.255533854166666}}}
#metrics {"StartTime": 1655370517.810209, "EndTime": 1655370517.8102183, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.631469128178615, "count": 1, "min": 14.631469128178615, "max": 14.631469128178615}}}
#metrics {"StartTime": 1655370517.810253, "EndTime": 1655370517.8102622, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.486032523360906, "count": 1, "min": 14.486032523360906, "max": 14.486032523360906}}}
#metrics {"StartTime": 1655370517.8102968, "EndTime": 1655370517.8103056, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.717605890012255, "count": 1, "min": 17.717605890012255, "max": 17.717605890012255}}}
#metrics {"StartTime": 1655370517.8103495, "EndTime": 1655370517.8103595, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 18.296094707414216, "count": 1, "min": 18.296094707414216, "max": 18.296094707414216}}}
#metrics {"StartTime": 1655370517.8103933, "EndTime": 1655370517.8104043, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.732822193818933, "count": 1, "min": 17.732822193818933, "max": 17.732822193818933}}}
#metrics {"StartTime": 1655370517.8104389, "EndTime": 1655370517.810449, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.891299378638173, "count": 1, "min": 17.891299378638173, "max": 17.891299378638173}}}
#metrics {"StartTime": 1655370517.8104844, "EndTime": 1655370517.8104947, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.400882496553308, "count": 1, "min": 19.400882496553308, "max": 19.400882496553308}}}
#metrics {"StartTime": 1655370517.8105319, "EndTime": 1655370517.810542, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.12230607575061, "count": 1, "min": 19.12230607575061, "max": 19.12230607575061}}}
#metrics {"StartTime": 1655370517.8105783, "EndTime": 1655370517.8105884, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.357296214384192, "count": 1, "min": 19.357296214384192, "max": 19.357296214384192}}}
#metrics {"StartTime": 1655370517.8106284, "EndTime": 1655370517.8106377, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.472297818053004, "count": 1, "min": 19.472297818053004, "max": 19.472297818053004}}}
#metrics {"StartTime": 1655370517.8106725, "EndTime": 1655370517.8106823, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.13913143382353, "count": 1, "min": 61.13913143382353, "max": 61.13913143382353}}}
#metrics {"StartTime": 1655370517.8107197, "EndTime": 1655370517.8107295, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 61.21798167509191, "count": 1, "min": 61.21798167509191, "max": 61.21798167509191}}}
#metrics {"StartTime": 1655370517.8107648, "EndTime": 1655370517.810774, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.512032303155635, "count": 1, "min": 61.512032303155635, "max": 61.512032303155635}}}
#metrics {"StartTime": 1655370517.8108106, "EndTime": 1655370517.8108199, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.2959534888174, "count": 1, "min": 61.2959534888174, "max": 61.2959534888174}}}
#metrics {"StartTime": 1655370517.8108575, "EndTime": 1655370517.8108675, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.87495212928921, "count": 1, "min": 61.87495212928921, "max": 61.87495212928921}}}
#metrics {"StartTime": 1655370517.8109033, "EndTime": 1655370517.8109133, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 62.793772977941174, "count": 1, "min": 62.793772977941174, "max": 62.793772977941174}}}
#metrics {"StartTime": 1655370517.81095, "EndTime": 1655370517.8109605, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 62.707282571231616, "count": 1, "min": 62.707282571231616, "max": 62.707282571231616}}}
#metrics {"StartTime": 1655370517.8109972, "EndTime": 1655370517.8110077, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.59827138863358, "count": 1, "min": 62.59827138863358, "max": 62.59827138863358}}}
[06/16/2022 09:08:37 INFO 139637139089216] #quality_metric: host=algo-1, epoch=5, validation mse_objective <loss>=14.737356148514094
[06/16/2022 09:08:37 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=5, criteria=mse_objective, value=13.732995725145527
[06/16/2022 09:08:37 INFO 139637139089216] Epoch 5: Loss has not improved for 0 epochs.
[06/16/2022 09:08:37 INFO 139637139089216] Saving model for epoch: 5
[06/16/2022 09:08:37 INFO 139637139089216] Saved checkpoint to "/tmp/tmpw6ev9kkq/mx-mod-0000.params"
[06/16/2022 09:08:37 INFO 139637139089216] #progress_metric: host=algo-1, completed 40.0 % of epochs
#metrics {"StartTime": 1655370517.0050483, "EndTime": 1655370517.819513, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 5, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 3215.0, "count": 1, "min": 3215, "max": 3215}, "Total Batches Seen": {"sum": 113.0, "count": 1, "min": 113, "max": 113}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 8.0, "count": 1, "min": 8, "max": 8}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:37 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=558.5615843364608 records/second
#metrics {"StartTime": 1655370518.4998982, "EndTime": 1655370518.4999816, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.30897735542721216, "count": 1, "min": 0.30897735542721216, "max": 0.30897735542721216}}}
#metrics {"StartTime": 1655370518.5000718, "EndTime": 1655370518.5000858, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.3147106679280599, "count": 1, "min": 0.3147106679280599, "max": 0.3147106679280599}}}
#metrics {"StartTime": 1655370518.5018978, "EndTime": 1655370518.5019257, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.32191754553053115, "count": 1, "min": 0.32191754553053115, "max": 0.32191754553053115}}}
#metrics {"StartTime": 1655370518.5021906, "EndTime": 1655370518.5022106, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.3126039653354221, "count": 1, "min": 0.3126039653354221, "max": 0.3126039653354221}}}
#metrics {"StartTime": 1655370518.502449, "EndTime": 1655370518.5024726, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.298829345703125, "count": 1, "min": 0.298829345703125, "max": 0.298829345703125}}}
#metrics {"StartTime": 1655370518.502766, "EndTime": 1655370518.502787, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.2972593699561225, "count": 1, "min": 0.2972593699561225, "max": 0.2972593699561225}}}
#metrics {"StartTime": 1655370518.5042772, "EndTime": 1655370518.5043008, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.29779706319173177, "count": 1, "min": 0.29779706319173177, "max": 0.29779706319173177}}}
#metrics {"StartTime": 1655370518.5045617, "EndTime": 1655370518.5045803, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.2949141089121501, "count": 1, "min": 0.2949141089121501, "max": 0.2949141089121501}}}
#metrics {"StartTime": 1655370518.504806, "EndTime": 1655370518.5048285, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.3098793829811944, "count": 1, "min": 0.3098793829811944, "max": 0.3098793829811944}}}
#metrics {"StartTime": 1655370518.5051146, "EndTime": 1655370518.505134, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.2965576050016615, "count": 1, "min": 0.2965576050016615, "max": 0.2965576050016615}}}
#metrics {"StartTime": 1655370518.5058932, "EndTime": 1655370518.5059142, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.3242960876888699, "count": 1, "min": 0.3242960876888699, "max": 0.3242960876888699}}}
#metrics {"StartTime": 1655370518.5061612, "EndTime": 1655370518.5061798, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.31914955139160156, "count": 1, "min": 0.31914955139160156, "max": 0.31914955139160156}}}
#metrics {"StartTime": 1655370518.5064259, "EndTime": 1655370518.506444, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.3023216772079468, "count": 1, "min": 0.3023216772079468, "max": 0.3023216772079468}}}
#metrics {"StartTime": 1655370518.5066624, "EndTime": 1655370518.5066795, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.30320469750298396, "count": 1, "min": 0.30320469750298396, "max": 0.30320469750298396}}}
#metrics {"StartTime": 1655370518.507678, "EndTime": 1655370518.5077007, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.2953969711727566, "count": 1, "min": 0.2953969711727566, "max": 0.2953969711727566}}}
#metrics {"StartTime": 1655370518.5077674, "EndTime": 1655370518.507781, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.2959550454881456, "count": 1, "min": 0.2959550454881456, "max": 0.2959550454881456}}}
#metrics {"StartTime": 1655370518.507822, "EndTime": 1655370518.5078332, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.3850594668918186, "count": 1, "min": 0.3850594668918186, "max": 0.3850594668918186}}}
#metrics {"StartTime": 1655370518.5079374, "EndTime": 1655370518.5079515, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.3889405075709025, "count": 1, "min": 0.3889405075709025, "max": 0.3889405075709025}}}
#metrics {"StartTime": 1655370518.5079908, "EndTime": 1655370518.508001, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.3830383904774984, "count": 1, "min": 0.3830383904774984, "max": 0.3830383904774984}}}
#metrics {"StartTime": 1655370518.5082567, "EndTime": 1655370518.508276, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.38429128328959145, "count": 1, "min": 0.38429128328959145, "max": 0.38429128328959145}}}
#metrics {"StartTime": 1655370518.508372, "EndTime": 1655370518.5083902, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.4012879572974311, "count": 1, "min": 0.4012879572974311, "max": 0.4012879572974311}}}
#metrics {"StartTime": 1655370518.5084398, "EndTime": 1655370518.5084553, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.40971195538838706, "count": 1, "min": 0.40971195538838706, "max": 0.40971195538838706}}}
#metrics {"StartTime": 1655370518.5085006, "EndTime": 1655370518.508511, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4050375186072456, "count": 1, "min": 0.4050375186072456, "max": 0.4050375186072456}}}
#metrics {"StartTime": 1655370518.5085492, "EndTime": 1655370518.5085595, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.4043354680803087, "count": 1, "min": 0.4043354680803087, "max": 0.4043354680803087}}}
#metrics {"StartTime": 1655370518.508597, "EndTime": 1655370518.5086062, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9556216049194336, "count": 1, "min": 0.9556216049194336, "max": 0.9556216049194336}}}
#metrics {"StartTime": 1655370518.508642, "EndTime": 1655370518.508652, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9563503032260471, "count": 1, "min": 0.9563503032260471, "max": 0.9563503032260471}}}
#metrics {"StartTime": 1655370518.508688, "EndTime": 1655370518.5086977, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9559640820821126, "count": 1, "min": 0.9559640820821126, "max": 0.9559640820821126}}}
#metrics {"StartTime": 1655370518.5087314, "EndTime": 1655370518.508741, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9559990077548557, "count": 1, "min": 0.9559990077548557, "max": 0.9559990077548557}}}
#metrics {"StartTime": 1655370518.5091834, "EndTime": 1655370518.509203, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9691147549947103, "count": 1, "min": 0.9691147549947103, "max": 0.9691147549947103}}}
#metrics {"StartTime": 1655370518.5092757, "EndTime": 1655370518.509291, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9743991894192165, "count": 1, "min": 0.9743991894192165, "max": 0.9743991894192165}}}
#metrics {"StartTime": 1655370518.5093346, "EndTime": 1655370518.5093465, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9726091236538357, "count": 1, "min": 0.9726091236538357, "max": 0.9726091236538357}}}
#metrics {"StartTime": 1655370518.509553, "EndTime": 1655370518.5095723, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9732530445522732, "count": 1, "min": 0.9732530445522732, "max": 0.9732530445522732}}}
[06/16/2022 09:08:38 INFO 139637139089216] #quality_metric: host=algo-1, epoch=6, train mse_objective <loss>=0.30897735542721216
#metrics {"StartTime": 1655370518.6367745, "EndTime": 1655370518.6368623, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 14.250956217447916, "count": 1, "min": 14.250956217447916, "max": 14.250956217447916}}}
#metrics {"StartTime": 1655370518.6369643, "EndTime": 1655370518.6369798, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 14.689021690219056, "count": 1, "min": 14.689021690219056, "max": 14.689021690219056}}}
#metrics {"StartTime": 1655370518.6370223, "EndTime": 1655370518.6370323, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 14.083184335746017, "count": 1, "min": 14.083184335746017, "max": 14.083184335746017}}}
#metrics {"StartTime": 1655370518.6370666, "EndTime": 1655370518.6370754, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 13.72221284754136, "count": 1, "min": 13.72221284754136, "max": 13.72221284754136}}}
#metrics {"StartTime": 1655370518.6371083, "EndTime": 1655370518.6371167, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.689798392501531, "count": 1, "min": 14.689798392501531, "max": 14.689798392501531}}}
#metrics {"StartTime": 1655370518.6386778, "EndTime": 1655370518.6387045, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.515152276731005, "count": 1, "min": 14.515152276731005, "max": 14.515152276731005}}}
#metrics {"StartTime": 1655370518.6387596, "EndTime": 1655370518.638771, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.671085731655944, "count": 1, "min": 14.671085731655944, "max": 14.671085731655944}}}
#metrics {"StartTime": 1655370518.6389523, "EndTime": 1655370518.6389682, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.720938888250613, "count": 1, "min": 14.720938888250613, "max": 14.720938888250613}}}
#metrics {"StartTime": 1655370518.639088, "EndTime": 1655370518.6391027, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 13.692630543428308, "count": 1, "min": 13.692630543428308, "max": 13.692630543428308}}}
#metrics {"StartTime": 1655370518.6392162, "EndTime": 1655370518.639231, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.588758281632966, "count": 1, "min": 13.588758281632966, "max": 13.588758281632966}}}
#metrics {"StartTime": 1655370518.639336, "EndTime": 1655370518.6393514, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 14.579924340341606, "count": 1, "min": 14.579924340341606, "max": 14.579924340341606}}}
#metrics {"StartTime": 1655370518.6393907, "EndTime": 1655370518.6394925, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 14.275383444393382, "count": 1, "min": 14.275383444393382, "max": 14.275383444393382}}}
#metrics {"StartTime": 1655370518.63954, "EndTime": 1655370518.6395514, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.390708773743873, "count": 1, "min": 14.390708773743873, "max": 14.390708773743873}}}
#metrics {"StartTime": 1655370518.6396685, "EndTime": 1655370518.6396823, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 14.988318349800858, "count": 1, "min": 14.988318349800858, "max": 14.988318349800858}}}
#metrics {"StartTime": 1655370518.6397872, "EndTime": 1655370518.6398005, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.409825343711704, "count": 1, "min": 14.409825343711704, "max": 14.409825343711704}}}
#metrics {"StartTime": 1655370518.6399052, "EndTime": 1655370518.6399183, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.56963991651348, "count": 1, "min": 14.56963991651348, "max": 14.56963991651348}}}
#metrics {"StartTime": 1655370518.640026, "EndTime": 1655370518.6400397, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.6280044854856, "count": 1, "min": 17.6280044854856, "max": 17.6280044854856}}}
#metrics {"StartTime": 1655370518.6401417, "EndTime": 1655370518.6401556, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 17.974991143918505, "count": 1, "min": 17.974991143918505, "max": 17.974991143918505}}}
#metrics {"StartTime": 1655370518.6402678, "EndTime": 1655370518.6403284, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.605397542317707, "count": 1, "min": 17.605397542317707, "max": 17.605397542317707}}}
#metrics {"StartTime": 1655370518.6403785, "EndTime": 1655370518.6403913, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.694723091873467, "count": 1, "min": 17.694723091873467, "max": 17.694723091873467}}}
#metrics {"StartTime": 1655370518.6404295, "EndTime": 1655370518.6404405, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.302534514782476, "count": 1, "min": 19.302534514782476, "max": 19.302534514782476}}}
#metrics {"StartTime": 1655370518.6405933, "EndTime": 1655370518.640609, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.0399564855239, "count": 1, "min": 19.0399564855239, "max": 19.0399564855239}}}
#metrics {"StartTime": 1655370518.6407275, "EndTime": 1655370518.6407945, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.32546398686428, "count": 1, "min": 19.32546398686428, "max": 19.32546398686428}}}
#metrics {"StartTime": 1655370518.6408415, "EndTime": 1655370518.6409123, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.384449079924938, "count": 1, "min": 19.384449079924938, "max": 19.384449079924938}}}
#metrics {"StartTime": 1655370518.6409583, "EndTime": 1655370518.64097, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.26442105162378, "count": 1, "min": 61.26442105162378, "max": 61.26442105162378}}}
#metrics {"StartTime": 1655370518.641084, "EndTime": 1655370518.6411507, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 61.05339020373774, "count": 1, "min": 61.05339020373774, "max": 61.05339020373774}}}
#metrics {"StartTime": 1655370518.6411977, "EndTime": 1655370518.64121, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.49389169730392, "count": 1, "min": 61.49389169730392, "max": 61.49389169730392}}}
#metrics {"StartTime": 1655370518.6413193, "EndTime": 1655370518.641358, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.271278530943626, "count": 1, "min": 61.271278530943626, "max": 61.271278530943626}}}
#metrics {"StartTime": 1655370518.6414037, "EndTime": 1655370518.6414168, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.872628006280635, "count": 1, "min": 61.872628006280635, "max": 61.872628006280635}}}
#metrics {"StartTime": 1655370518.6414537, "EndTime": 1655370518.6414974, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 61.867237764246326, "count": 1, "min": 61.867237764246326, "max": 61.867237764246326}}}
#metrics {"StartTime": 1655370518.6415405, "EndTime": 1655370518.6415534, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 62.593525007659316, "count": 1, "min": 62.593525007659316, "max": 62.593525007659316}}}
#metrics {"StartTime": 1655370518.641591, "EndTime": 1655370518.641602, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.01347321155024, "count": 1, "min": 62.01347321155024, "max": 62.01347321155024}}}
[06/16/2022 09:08:38 INFO 139637139089216] #quality_metric: host=algo-1, epoch=6, validation mse_objective <loss>=14.250956217447916
[06/16/2022 09:08:38 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=6, criteria=mse_objective, value=13.588758281632966
[06/16/2022 09:08:38 INFO 139637139089216] Epoch 6: Loss has not improved for 0 epochs.
[06/16/2022 09:08:38 INFO 139637139089216] Saving model for epoch: 6
[06/16/2022 09:08:38 INFO 139637139089216] Saved checkpoint to "/tmp/tmp8mtyghhx/mx-mod-0000.params"
[06/16/2022 09:08:38 INFO 139637139089216] #progress_metric: host=algo-1, completed 46.666666666666664 % of epochs
#metrics {"StartTime": 1655370517.8198147, "EndTime": 1655370518.654596, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 6, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 3670.0, "count": 1, "min": 3670, "max": 3670}, "Total Batches Seen": {"sum": 129.0, "count": 1, "min": 129, "max": 129}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 9.0, "count": 1, "min": 9, "max": 9}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:38 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=544.9471874120147 records/second
#metrics {"StartTime": 1655370519.4299972, "EndTime": 1655370519.430078, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.30013485908508303, "count": 1, "min": 0.30013485908508303, "max": 0.30013485908508303}}}
#metrics {"StartTime": 1655370519.4301653, "EndTime": 1655370519.4301777, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.30465739727020263, "count": 1, "min": 0.30465739727020263, "max": 0.30465739727020263}}}
#metrics {"StartTime": 1655370519.4302166, "EndTime": 1655370519.4302263, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.3082283205456204, "count": 1, "min": 0.3082283205456204, "max": 0.3082283205456204}}}
#metrics {"StartTime": 1655370519.4302602, "EndTime": 1655370519.4302688, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.3010712273915609, "count": 1, "min": 0.3010712273915609, "max": 0.3010712273915609}}}
#metrics {"StartTime": 1655370519.4303105, "EndTime": 1655370519.4303195, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.2989350361294217, "count": 1, "min": 0.2989350361294217, "max": 0.2989350361294217}}}
#metrics {"StartTime": 1655370519.4303935, "EndTime": 1655370519.4304044, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.29621562216016983, "count": 1, "min": 0.29621562216016983, "max": 0.29621562216016983}}}
#metrics {"StartTime": 1655370519.4304388, "EndTime": 1655370519.430448, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.2971478091345893, "count": 1, "min": 0.2971478091345893, "max": 0.2971478091345893}}}
#metrics {"StartTime": 1655370519.4304814, "EndTime": 1655370519.4304914, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.2961660967932807, "count": 1, "min": 0.2961660967932807, "max": 0.2961660967932807}}}
#metrics {"StartTime": 1655370519.4305267, "EndTime": 1655370519.4305365, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.30016189840104845, "count": 1, "min": 0.30016189840104845, "max": 0.30016189840104845}}}
#metrics {"StartTime": 1655370519.430663, "EndTime": 1655370519.430692, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.2906233353084988, "count": 1, "min": 0.2906233353084988, "max": 0.2906233353084988}}}
#metrics {"StartTime": 1655370519.4307392, "EndTime": 1655370519.4307497, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.31111229684617786, "count": 1, "min": 0.31111229684617786, "max": 0.31111229684617786}}}
#metrics {"StartTime": 1655370519.4307883, "EndTime": 1655370519.4307983, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.3052790975570679, "count": 1, "min": 0.3052790975570679, "max": 0.3052790975570679}}}
#metrics {"StartTime": 1655370519.4308348, "EndTime": 1655370519.4308448, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.29876067373487686, "count": 1, "min": 0.29876067373487686, "max": 0.29876067373487686}}}
#metrics {"StartTime": 1655370519.4308815, "EndTime": 1655370519.4308918, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.3051757060156928, "count": 1, "min": 0.3051757060156928, "max": 0.3051757060156928}}}
#metrics {"StartTime": 1655370519.4309273, "EndTime": 1655370519.430937, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.29522416432698567, "count": 1, "min": 0.29522416432698567, "max": 0.29522416432698567}}}
#metrics {"StartTime": 1655370519.4310412, "EndTime": 1655370519.431064, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.2963858434889052, "count": 1, "min": 0.2963858434889052, "max": 0.2963858434889052}}}
#metrics {"StartTime": 1655370519.431106, "EndTime": 1655370519.431116, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.3823889128367106, "count": 1, "min": 0.3823889128367106, "max": 0.3823889128367106}}}
#metrics {"StartTime": 1655370519.4311528, "EndTime": 1655370519.4311635, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.38452766100565594, "count": 1, "min": 0.38452766100565594, "max": 0.38452766100565594}}}
#metrics {"StartTime": 1655370519.4312005, "EndTime": 1655370519.4312105, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.3809778838687473, "count": 1, "min": 0.3809778838687473, "max": 0.3809778838687473}}}
#metrics {"StartTime": 1655370519.431247, "EndTime": 1655370519.4312565, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.3815756458706326, "count": 1, "min": 0.3815756458706326, "max": 0.3815756458706326}}}
#metrics {"StartTime": 1655370519.4312935, "EndTime": 1655370519.4313033, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.40013744566175674, "count": 1, "min": 0.40013744566175674, "max": 0.40013744566175674}}}
#metrics {"StartTime": 1655370519.431381, "EndTime": 1655370519.4314406, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.41109680705600316, "count": 1, "min": 0.41109680705600316, "max": 0.41109680705600316}}}
#metrics {"StartTime": 1655370519.4314873, "EndTime": 1655370519.4314997, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4048978000217014, "count": 1, "min": 0.4048978000217014, "max": 0.4048978000217014}}}
#metrics {"StartTime": 1655370519.4315376, "EndTime": 1655370519.4315474, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.4058122613694933, "count": 1, "min": 0.4058122613694933, "max": 0.4058122613694933}}}
#metrics {"StartTime": 1655370519.4315846, "EndTime": 1655370519.4315948, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9561879285176595, "count": 1, "min": 0.9561879285176595, "max": 0.9561879285176595}}}
#metrics {"StartTime": 1655370519.4316313, "EndTime": 1655370519.431641, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9562840292188857, "count": 1, "min": 0.9562840292188857, "max": 0.9562840292188857}}}
#metrics {"StartTime": 1655370519.4317672, "EndTime": 1655370519.4317803, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9554361216227214, "count": 1, "min": 0.9554361216227214, "max": 0.9554361216227214}}}
#metrics {"StartTime": 1655370519.431821, "EndTime": 1655370519.431832, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9560291735331218, "count": 1, "min": 0.9560291735331218, "max": 0.9560291735331218}}}
#metrics {"StartTime": 1655370519.431869, "EndTime": 1655370519.4318786, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9693400192260743, "count": 1, "min": 0.9693400192260743, "max": 0.9693400192260743}}}
#metrics {"StartTime": 1655370519.431916, "EndTime": 1655370519.4319263, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.973510250515408, "count": 1, "min": 0.973510250515408, "max": 0.973510250515408}}}
#metrics {"StartTime": 1655370519.4320345, "EndTime": 1655370519.4320471, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9729773924085829, "count": 1, "min": 0.9729773924085829, "max": 0.9729773924085829}}}
#metrics {"StartTime": 1655370519.4320884, "EndTime": 1655370519.432099, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9738874668545193, "count": 1, "min": 0.9738874668545193, "max": 0.9738874668545193}}}
[06/16/2022 09:08:39 INFO 139637139089216] #quality_metric: host=algo-1, epoch=7, train mse_objective <loss>=0.30013485908508303
#metrics {"StartTime": 1655370519.5213199, "EndTime": 1655370519.5214026, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 14.000498453776041, "count": 1, "min": 14.000498453776041, "max": 14.000498453776041}}}
#metrics {"StartTime": 1655370519.521495, "EndTime": 1655370519.5215108, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 14.325359269684435, "count": 1, "min": 14.325359269684435, "max": 14.325359269684435}}}
#metrics {"StartTime": 1655370519.521554, "EndTime": 1655370519.521565, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 13.608120188993567, "count": 1, "min": 13.608120188993567, "max": 13.608120188993567}}}
#metrics {"StartTime": 1655370519.5215993, "EndTime": 1655370519.5216086, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 13.467373717064952, "count": 1, "min": 13.467373717064952, "max": 13.467373717064952}}}
#metrics {"StartTime": 1655370519.523497, "EndTime": 1655370519.5235257, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.512777889476103, "count": 1, "min": 14.512777889476103, "max": 14.512777889476103}}}
#metrics {"StartTime": 1655370519.523622, "EndTime": 1655370519.5236387, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.480385574640012, "count": 1, "min": 14.480385574640012, "max": 14.480385574640012}}}
#metrics {"StartTime": 1655370519.5236857, "EndTime": 1655370519.5236971, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.48170441272212, "count": 1, "min": 14.48170441272212, "max": 14.48170441272212}}}
#metrics {"StartTime": 1655370519.5237367, "EndTime": 1655370519.5237474, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.422599044500613, "count": 1, "min": 14.422599044500613, "max": 14.422599044500613}}}
#metrics {"StartTime": 1655370519.5237856, "EndTime": 1655370519.5237954, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 13.470084994446998, "count": 1, "min": 13.470084994446998, "max": 13.470084994446998}}}
#metrics {"StartTime": 1655370519.5238717, "EndTime": 1655370519.523903, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.578092687270221, "count": 1, "min": 13.578092687270221, "max": 13.578092687270221}}}
#metrics {"StartTime": 1655370519.5239477, "EndTime": 1655370519.5239587, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 14.167259066712623, "count": 1, "min": 14.167259066712623, "max": 14.167259066712623}}}
#metrics {"StartTime": 1655370519.5239959, "EndTime": 1655370519.5240064, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 13.755745083678002, "count": 1, "min": 13.755745083678002, "max": 13.755745083678002}}}
#metrics {"StartTime": 1655370519.5240428, "EndTime": 1655370519.524087, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.471944173177084, "count": 1, "min": 14.471944173177084, "max": 14.471944173177084}}}
#metrics {"StartTime": 1655370519.5241642, "EndTime": 1655370519.5241761, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 14.646896063112745, "count": 1, "min": 14.646896063112745, "max": 14.646896063112745}}}
#metrics {"StartTime": 1655370519.5242143, "EndTime": 1655370519.5242238, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.481415991689644, "count": 1, "min": 14.481415991689644, "max": 14.481415991689644}}}
#metrics {"StartTime": 1655370519.5242627, "EndTime": 1655370519.5242732, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.378122965494791, "count": 1, "min": 14.378122965494791, "max": 14.378122965494791}}}
#metrics {"StartTime": 1655370519.5243645, "EndTime": 1655370519.5243778, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.55825985179228, "count": 1, "min": 17.55825985179228, "max": 17.55825985179228}}}
#metrics {"StartTime": 1655370519.5244193, "EndTime": 1655370519.5244296, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 17.773896460439644, "count": 1, "min": 17.773896460439644, "max": 17.773896460439644}}}
#metrics {"StartTime": 1655370519.5244668, "EndTime": 1655370519.524477, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.5570894129136, "count": 1, "min": 17.5570894129136, "max": 17.5570894129136}}}
#metrics {"StartTime": 1655370519.5245142, "EndTime": 1655370519.524524, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.60993089862898, "count": 1, "min": 17.60993089862898, "max": 17.60993089862898}}}
#metrics {"StartTime": 1655370519.5246346, "EndTime": 1655370519.524648, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.35096471449908, "count": 1, "min": 19.35096471449908, "max": 19.35096471449908}}}
#metrics {"StartTime": 1655370519.5246878, "EndTime": 1655370519.5246992, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.111102534275428, "count": 1, "min": 19.111102534275428, "max": 19.111102534275428}}}
#metrics {"StartTime": 1655370519.524736, "EndTime": 1655370519.5247462, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.373234767539827, "count": 1, "min": 19.373234767539827, "max": 19.373234767539827}}}
#metrics {"StartTime": 1655370519.5247831, "EndTime": 1655370519.5247931, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.383114683861827, "count": 1, "min": 19.383114683861827, "max": 19.383114683861827}}}
#metrics {"StartTime": 1655370519.5249014, "EndTime": 1655370519.5249147, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.15340887331495, "count": 1, "min": 61.15340887331495, "max": 61.15340887331495}}}
#metrics {"StartTime": 1655370519.5249536, "EndTime": 1655370519.5249636, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 61.030311734068626, "count": 1, "min": 61.030311734068626, "max": 61.030311734068626}}}
#metrics {"StartTime": 1655370519.5250013, "EndTime": 1655370519.5250118, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.46714872472427, "count": 1, "min": 61.46714872472427, "max": 61.46714872472427}}}
#metrics {"StartTime": 1655370519.5250814, "EndTime": 1655370519.525126, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.247446097579655, "count": 1, "min": 61.247446097579655, "max": 61.247446097579655}}}
#metrics {"StartTime": 1655370519.5251713, "EndTime": 1655370519.525183, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.59733790977329, "count": 1, "min": 61.59733790977329, "max": 61.59733790977329}}}
#metrics {"StartTime": 1655370519.5252213, "EndTime": 1655370519.5252318, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 61.56002508425245, "count": 1, "min": 61.56002508425245, "max": 61.56002508425245}}}
#metrics {"StartTime": 1655370519.5252697, "EndTime": 1655370519.5252807, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 62.363518210018384, "count": 1, "min": 62.363518210018384, "max": 62.363518210018384}}}
#metrics {"StartTime": 1655370519.5253673, "EndTime": 1655370519.5253975, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.14849973192402, "count": 1, "min": 62.14849973192402, "max": 62.14849973192402}}}
[06/16/2022 09:08:39 INFO 139637139089216] #quality_metric: host=algo-1, epoch=7, validation mse_objective <loss>=14.000498453776041
[06/16/2022 09:08:39 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=7, criteria=mse_objective, value=13.467373717064952
[06/16/2022 09:08:39 INFO 139637139089216] Epoch 7: Loss has not improved for 0 epochs.
[06/16/2022 09:08:39 INFO 139637139089216] Saving model for epoch: 7
[06/16/2022 09:08:39 INFO 139637139089216] Saved checkpoint to "/tmp/tmpn_4ygrzt/mx-mod-0000.params"
[06/16/2022 09:08:39 INFO 139637139089216] #progress_metric: host=algo-1, completed 53.333333333333336 % of epochs
#metrics {"StartTime": 1655370518.6549575, "EndTime": 1655370519.5368323, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 7, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 4125.0, "count": 1, "min": 4125, "max": 4125}, "Total Batches Seen": {"sum": 145.0, "count": 1, "min": 145, "max": 145}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 10.0, "count": 1, "min": 10, "max": 10}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:39 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=514.8640084000922 records/second
#metrics {"StartTime": 1655370520.0116363, "EndTime": 1655370520.0116863, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.2941920460595025, "count": 1, "min": 0.2941920460595025, "max": 0.2941920460595025}}}
#metrics {"StartTime": 1655370520.011752, "EndTime": 1655370520.0117636, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.2977325047387017, "count": 1, "min": 0.2977325047387017, "max": 0.2977325047387017}}}
#metrics {"StartTime": 1655370520.0118022, "EndTime": 1655370520.0118113, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.299101005660163, "count": 1, "min": 0.299101005660163, "max": 0.299101005660163}}}
#metrics {"StartTime": 1655370520.0118444, "EndTime": 1655370520.0118527, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.29377426359388564, "count": 1, "min": 0.29377426359388564, "max": 0.29377426359388564}}}
#metrics {"StartTime": 1655370520.0118868, "EndTime": 1655370520.0118957, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.2996904797024197, "count": 1, "min": 0.2996904797024197, "max": 0.2996904797024197}}}
#metrics {"StartTime": 1655370520.0119596, "EndTime": 1655370520.011973, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.29708516014946834, "count": 1, "min": 0.29708516014946834, "max": 0.29708516014946834}}}
#metrics {"StartTime": 1655370520.0120091, "EndTime": 1655370520.0120187, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.2980598428514269, "count": 1, "min": 0.2980598428514269, "max": 0.2980598428514269}}}
#metrics {"StartTime": 1655370520.0120518, "EndTime": 1655370520.0120609, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.29740745650397404, "count": 1, "min": 0.29740745650397404, "max": 0.29740745650397404}}}
#metrics {"StartTime": 1655370520.0120983, "EndTime": 1655370520.0121088, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.2937090386284722, "count": 1, "min": 0.2937090386284722, "max": 0.2937090386284722}}}
#metrics {"StartTime": 1655370520.0121424, "EndTime": 1655370520.012152, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.2868423456615872, "count": 1, "min": 0.2868423456615872, "max": 0.2868423456615872}}}
#metrics {"StartTime": 1655370520.0122473, "EndTime": 1655370520.0122707, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.3020733462439643, "count": 1, "min": 0.3020733462439643, "max": 0.3020733462439643}}}
#metrics {"StartTime": 1655370520.012336, "EndTime": 1655370520.0123484, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.29631993611653645, "count": 1, "min": 0.29631993611653645, "max": 0.29631993611653645}}}
#metrics {"StartTime": 1655370520.0123844, "EndTime": 1655370520.0123942, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.2968277194764879, "count": 1, "min": 0.2968277194764879, "max": 0.2968277194764879}}}
#metrics {"StartTime": 1655370520.012429, "EndTime": 1655370520.0124393, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.3076706706153022, "count": 1, "min": 0.3076706706153022, "max": 0.3076706706153022}}}
#metrics {"StartTime": 1655370520.0124753, "EndTime": 1655370520.0124848, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.2947852929433187, "count": 1, "min": 0.2947852929433187, "max": 0.2947852929433187}}}
#metrics {"StartTime": 1655370520.01252, "EndTime": 1655370520.0125296, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.29740756352742515, "count": 1, "min": 0.29740756352742515, "max": 0.29740756352742515}}}
#metrics {"StartTime": 1655370520.0126102, "EndTime": 1655370520.0126433, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.38085482279459637, "count": 1, "min": 0.38085482279459637, "max": 0.38085482279459637}}}
#metrics {"StartTime": 1655370520.0126982, "EndTime": 1655370520.0127106, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.3819334877861871, "count": 1, "min": 0.3819334877861871, "max": 0.3819334877861871}}}
#metrics {"StartTime": 1655370520.01275, "EndTime": 1655370520.0127604, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.38018073505825467, "count": 1, "min": 0.38018073505825467, "max": 0.38018073505825467}}}
#metrics {"StartTime": 1655370520.0127964, "EndTime": 1655370520.0128067, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.3805082925160726, "count": 1, "min": 0.3805082925160726, "max": 0.3805082925160726}}}
#metrics {"StartTime": 1655370520.0128434, "EndTime": 1655370520.0128539, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.4009894773695204, "count": 1, "min": 0.4009894773695204, "max": 0.4009894773695204}}}
#metrics {"StartTime": 1655370520.0128891, "EndTime": 1655370520.0129316, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.4127677832709418, "count": 1, "min": 0.4127677832709418, "max": 0.4127677832709418}}}
#metrics {"StartTime": 1655370520.0130405, "EndTime": 1655370520.0130532, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4057879130045573, "count": 1, "min": 0.4057879130045573, "max": 0.4057879130045573}}}
#metrics {"StartTime": 1655370520.0130932, "EndTime": 1655370520.013104, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.40640818066067164, "count": 1, "min": 0.40640818066067164, "max": 0.40640818066067164}}}
#metrics {"StartTime": 1655370520.0131407, "EndTime": 1655370520.01315, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9558209673563639, "count": 1, "min": 0.9558209673563639, "max": 0.9558209673563639}}}
#metrics {"StartTime": 1655370520.013188, "EndTime": 1655370520.0131981, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9563378630744086, "count": 1, "min": 0.9563378630744086, "max": 0.9563378630744086}}}
#metrics {"StartTime": 1655370520.0133023, "EndTime": 1655370520.013317, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.955727530585395, "count": 1, "min": 0.955727530585395, "max": 0.955727530585395}}}
#metrics {"StartTime": 1655370520.0133593, "EndTime": 1655370520.01337, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9561675283643934, "count": 1, "min": 0.9561675283643934, "max": 0.9561675283643934}}}
#metrics {"StartTime": 1655370520.0134063, "EndTime": 1655370520.0134165, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9690417692396376, "count": 1, "min": 0.9690417692396376, "max": 0.9690417692396376}}}
#metrics {"StartTime": 1655370520.0134535, "EndTime": 1655370520.0134635, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9747900433010526, "count": 1, "min": 0.9747900433010526, "max": 0.9747900433010526}}}
#metrics {"StartTime": 1655370520.0135748, "EndTime": 1655370520.0135884, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9729332139756944, "count": 1, "min": 0.9729332139756944, "max": 0.9729332139756944}}}
#metrics {"StartTime": 1655370520.0136409, "EndTime": 1655370520.013653, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9750834634568957, "count": 1, "min": 0.9750834634568957, "max": 0.9750834634568957}}}
[06/16/2022 09:08:40 INFO 139637139089216] #quality_metric: host=algo-1, epoch=8, train mse_objective <loss>=0.2941920460595025
#metrics {"StartTime": 1655370520.0893464, "EndTime": 1655370520.0893962, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 13.891092337814031, "count": 1, "min": 13.891092337814031, "max": 13.891092337814031}}}
#metrics {"StartTime": 1655370520.0894663, "EndTime": 1655370520.0894792, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 14.127114688648897, "count": 1, "min": 14.127114688648897, "max": 14.127114688648897}}}
#metrics {"StartTime": 1655370520.0895193, "EndTime": 1655370520.0895288, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 13.375494863472733, "count": 1, "min": 13.375494863472733, "max": 13.375494863472733}}}
#metrics {"StartTime": 1655370520.0895627, "EndTime": 1655370520.0895712, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 13.390127742991728, "count": 1, "min": 13.390127742991728, "max": 13.390127742991728}}}
#metrics {"StartTime": 1655370520.0896049, "EndTime": 1655370520.089613, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.589366239659926, "count": 1, "min": 14.589366239659926, "max": 14.589366239659926}}}
#metrics {"StartTime": 1655370520.0911295, "EndTime": 1655370520.0911534, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.518709668926164, "count": 1, "min": 14.518709668926164, "max": 14.518709668926164}}}
#metrics {"StartTime": 1655370520.09121, "EndTime": 1655370520.0912225, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.604721966911764, "count": 1, "min": 14.604721966911764, "max": 14.604721966911764}}}
#metrics {"StartTime": 1655370520.0913134, "EndTime": 1655370520.0913277, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.566698261335784, "count": 1, "min": 14.566698261335784, "max": 14.566698261335784}}}
#metrics {"StartTime": 1655370520.0914166, "EndTime": 1655370520.0914292, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 13.38850492589614, "count": 1, "min": 13.38850492589614, "max": 13.38850492589614}}}
#metrics {"StartTime": 1655370520.0914688, "EndTime": 1655370520.091479, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.63548667758119, "count": 1, "min": 13.63548667758119, "max": 13.63548667758119}}}
#metrics {"StartTime": 1655370520.091515, "EndTime": 1655370520.0915265, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 13.932566923253676, "count": 1, "min": 13.932566923253676, "max": 13.932566923253676}}}
#metrics {"StartTime": 1655370520.0915627, "EndTime": 1655370520.0915728, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 13.517542820350796, "count": 1, "min": 13.517542820350796, "max": 13.517542820350796}}}
#metrics {"StartTime": 1655370520.0916889, "EndTime": 1655370520.0917022, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.374922210094976, "count": 1, "min": 14.374922210094976, "max": 14.374922210094976}}}
#metrics {"StartTime": 1655370520.0917413, "EndTime": 1655370520.0917516, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 14.99994016161152, "count": 1, "min": 14.99994016161152, "max": 14.99994016161152}}}
#metrics {"StartTime": 1655370520.091789, "EndTime": 1655370520.0917997, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.399116067325368, "count": 1, "min": 14.399116067325368, "max": 14.399116067325368}}}
#metrics {"StartTime": 1655370520.0918357, "EndTime": 1655370520.091881, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.498655431410846, "count": 1, "min": 14.498655431410846, "max": 14.498655431410846}}}
#metrics {"StartTime": 1655370520.0919576, "EndTime": 1655370520.0919702, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.53235102634804, "count": 1, "min": 17.53235102634804, "max": 17.53235102634804}}}
#metrics {"StartTime": 1655370520.0920084, "EndTime": 1655370520.0920186, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 17.663778267654717, "count": 1, "min": 17.663778267654717, "max": 17.663778267654717}}}
#metrics {"StartTime": 1655370520.0920556, "EndTime": 1655370520.092066, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.548595353668812, "count": 1, "min": 17.548595353668812, "max": 17.548595353668812}}}
#metrics {"StartTime": 1655370520.0921018, "EndTime": 1655370520.0921464, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.57577335133272, "count": 1, "min": 17.57577335133272, "max": 17.57577335133272}}}
#metrics {"StartTime": 1655370520.0922227, "EndTime": 1655370520.0922353, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.417114856196385, "count": 1, "min": 19.417114856196385, "max": 19.417114856196385}}}
#metrics {"StartTime": 1655370520.0922737, "EndTime": 1655370520.0922842, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.064445944393384, "count": 1, "min": 19.064445944393384, "max": 19.064445944393384}}}
#metrics {"StartTime": 1655370520.0923226, "EndTime": 1655370520.0923333, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.305278104894303, "count": 1, "min": 19.305278104894303, "max": 19.305278104894303}}}
#metrics {"StartTime": 1655370520.0924037, "EndTime": 1655370520.092449, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.309446447035846, "count": 1, "min": 19.309446447035846, "max": 19.309446447035846}}}
#metrics {"StartTime": 1655370520.092494, "EndTime": 1655370520.0925057, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.16492895986519, "count": 1, "min": 61.16492895986519, "max": 61.16492895986519}}}
#metrics {"StartTime": 1655370520.0925431, "EndTime": 1655370520.0925536, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 61.01647470511642, "count": 1, "min": 61.01647470511642, "max": 61.01647470511642}}}
#metrics {"StartTime": 1655370520.0925922, "EndTime": 1655370520.092603, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.45632515701593, "count": 1, "min": 61.45632515701593, "max": 61.45632515701593}}}
#metrics {"StartTime": 1655370520.0926373, "EndTime": 1655370520.0927045, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.2218807444853, "count": 1, "min": 61.2218807444853, "max": 61.2218807444853}}}
#metrics {"StartTime": 1655370520.092766, "EndTime": 1655370520.0927784, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.41029268152573, "count": 1, "min": 61.41029268152573, "max": 61.41029268152573}}}
#metrics {"StartTime": 1655370520.092816, "EndTime": 1655370520.0928268, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 61.66987400428921, "count": 1, "min": 61.66987400428921, "max": 61.66987400428921}}}
#metrics {"StartTime": 1655370520.0928633, "EndTime": 1655370520.092874, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 62.16052724800858, "count": 1, "min": 62.16052724800858, "max": 62.16052724800858}}}
#metrics {"StartTime": 1655370520.092909, "EndTime": 1655370520.0929692, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.23718501072304, "count": 1, "min": 62.23718501072304, "max": 62.23718501072304}}}
[06/16/2022 09:08:40 INFO 139637139089216] #quality_metric: host=algo-1, epoch=8, validation mse_objective <loss>=13.891092337814031
[06/16/2022 09:08:40 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=8, criteria=mse_objective, value=13.375494863472733
[06/16/2022 09:08:40 INFO 139637139089216] Epoch 8: Loss has not improved for 0 epochs.
[06/16/2022 09:08:40 INFO 139637139089216] Saving model for epoch: 8
[06/16/2022 09:08:40 INFO 139637139089216] Saved checkpoint to "/tmp/tmpnn4ry60m/mx-mod-0000.params"
[06/16/2022 09:08:40 INFO 139637139089216] #progress_metric: host=algo-1, completed 60.0 % of epochs
#metrics {"StartTime": 1655370519.5388684, "EndTime": 1655370520.10428, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 8, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 4580.0, "count": 1, "min": 4580, "max": 4580}, "Total Batches Seen": {"sum": 161.0, "count": 1, "min": 161, "max": 161}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 11.0, "count": 1, "min": 11, "max": 11}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:40 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=804.4782822583467 records/second
#metrics {"StartTime": 1655370520.549621, "EndTime": 1655370520.5497212, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.29013213634490964, "count": 1, "min": 0.29013213634490964, "max": 0.29013213634490964}}}
#metrics {"StartTime": 1655370520.5502884, "EndTime": 1655370520.5503304, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.2928758223851522, "count": 1, "min": 0.2928758223851522, "max": 0.2928758223851522}}}
#metrics {"StartTime": 1655370520.550614, "EndTime": 1655370520.5506585, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.2929053433736165, "count": 1, "min": 0.2929053433736165, "max": 0.2929053433736165}}}
#metrics {"StartTime": 1655370520.5509858, "EndTime": 1655370520.5510325, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.28909748130374485, "count": 1, "min": 0.28909748130374485, "max": 0.28909748130374485}}}
#metrics {"StartTime": 1655370520.5513449, "EndTime": 1655370520.5513887, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.30040884017944336, "count": 1, "min": 0.30040884017944336, "max": 0.30040884017944336}}}
#metrics {"StartTime": 1655370520.5516994, "EndTime": 1655370520.551744, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.2986269039577908, "count": 1, "min": 0.2986269039577908, "max": 0.2986269039577908}}}
#metrics {"StartTime": 1655370520.5520782, "EndTime": 1655370520.5521278, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.2992264938354492, "count": 1, "min": 0.2992264938354492, "max": 0.2992264938354492}}}
#metrics {"StartTime": 1655370520.5524397, "EndTime": 1655370520.5524619, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.29840770721435544, "count": 1, "min": 0.29840770721435544, "max": 0.29840770721435544}}}
#metrics {"StartTime": 1655370520.552744, "EndTime": 1655370520.5527666, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.28933507495456273, "count": 1, "min": 0.28933507495456273, "max": 0.28933507495456273}}}
#metrics {"StartTime": 1655370520.5530725, "EndTime": 1655370520.5530975, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.2842833508385552, "count": 1, "min": 0.2842833508385552, "max": 0.2842833508385552}}}
#metrics {"StartTime": 1655370520.553432, "EndTime": 1655370520.5534549, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.29578848520914713, "count": 1, "min": 0.29578848520914713, "max": 0.29578848520914713}}}
#metrics {"StartTime": 1655370520.5537934, "EndTime": 1655370520.5538404, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.2905331203672621, "count": 1, "min": 0.2905331203672621, "max": 0.2905331203672621}}}
#metrics {"StartTime": 1655370520.5541644, "EndTime": 1655370520.5542152, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.2965577051374647, "count": 1, "min": 0.2965577051374647, "max": 0.2965577051374647}}}
#metrics {"StartTime": 1655370520.5545385, "EndTime": 1655370520.554582, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.3103733253479004, "count": 1, "min": 0.3103733253479004, "max": 0.3103733253479004}}}
#metrics {"StartTime": 1655370520.5548851, "EndTime": 1655370520.5549314, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.2948703628116184, "count": 1, "min": 0.2948703628116184, "max": 0.2948703628116184}}}
#metrics {"StartTime": 1655370520.5552418, "EndTime": 1655370520.5552843, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.2986698521508111, "count": 1, "min": 0.2986698521508111, "max": 0.2986698521508111}}}
#metrics {"StartTime": 1655370520.5555859, "EndTime": 1655370520.555631, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.38023031128777396, "count": 1, "min": 0.38023031128777396, "max": 0.38023031128777396}}}
#metrics {"StartTime": 1655370520.5559533, "EndTime": 1655370520.5560029, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.3807586415608724, "count": 1, "min": 0.3807586415608724, "max": 0.3807586415608724}}}
#metrics {"StartTime": 1655370520.556318, "EndTime": 1655370520.5563595, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.37992733743455676, "count": 1, "min": 0.37992733743455676, "max": 0.37992733743455676}}}
#metrics {"StartTime": 1655370520.556667, "EndTime": 1655370520.556708, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.3801351261138916, "count": 1, "min": 0.3801351261138916, "max": 0.3801351261138916}}}
#metrics {"StartTime": 1655370520.5570157, "EndTime": 1655370520.5570614, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.4010053952534994, "count": 1, "min": 0.4010053952534994, "max": 0.4010053952534994}}}
#metrics {"StartTime": 1655370520.557363, "EndTime": 1655370520.5574064, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.4145890680948893, "count": 1, "min": 0.4145890680948893, "max": 0.4145890680948893}}}
#metrics {"StartTime": 1655370520.5577104, "EndTime": 1655370520.5577602, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4057956960466173, "count": 1, "min": 0.4057956960466173, "max": 0.4057956960466173}}}
#metrics {"StartTime": 1655370520.558101, "EndTime": 1655370520.5581424, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.4078213649325901, "count": 1, "min": 0.4078213649325901, "max": 0.4078213649325901}}}
#metrics {"StartTime": 1655370520.5584185, "EndTime": 1655370520.5584414, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9560589875115288, "count": 1, "min": 0.9560589875115288, "max": 0.9560589875115288}}}
#metrics {"StartTime": 1655370520.558794, "EndTime": 1655370520.5588386, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9563559150695801, "count": 1, "min": 0.9563559150695801, "max": 0.9563559150695801}}}
#metrics {"StartTime": 1655370520.559153, "EndTime": 1655370520.5591962, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9556234571668837, "count": 1, "min": 0.9556234571668837, "max": 0.9556234571668837}}}
#metrics {"StartTime": 1655370520.5594993, "EndTime": 1655370520.5595422, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9561930656433105, "count": 1, "min": 0.9561930656433105, "max": 0.9561930656433105}}}
#metrics {"StartTime": 1655370520.5598738, "EndTime": 1655370520.5599148, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9691415850321452, "count": 1, "min": 0.9691415850321452, "max": 0.9691415850321452}}}
#metrics {"StartTime": 1655370520.5601826, "EndTime": 1655370520.560206, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9761156145731608, "count": 1, "min": 0.9761156145731608, "max": 0.9761156145731608}}}
#metrics {"StartTime": 1655370520.5605092, "EndTime": 1655370520.5605328, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9731467416551378, "count": 1, "min": 0.9731467416551378, "max": 0.9731467416551378}}}
#metrics {"StartTime": 1655370520.5608394, "EndTime": 1655370520.5608828, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9760680919223361, "count": 1, "min": 0.9760680919223361, "max": 0.9760680919223361}}}
[06/16/2022 09:08:40 INFO 139637139089216] #quality_metric: host=algo-1, epoch=9, train mse_objective <loss>=0.29013213634490964
#metrics {"StartTime": 1655370520.6441615, "EndTime": 1655370520.6442583, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 13.868024040670957, "count": 1, "min": 13.868024040670957, "max": 13.868024040670957}}}
#metrics {"StartTime": 1655370520.6448584, "EndTime": 1655370520.6448953, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 14.04374246036305, "count": 1, "min": 14.04374246036305, "max": 14.04374246036305}}}
#metrics {"StartTime": 1655370520.6451862, "EndTime": 1655370520.6452343, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 13.290261062921262, "count": 1, "min": 13.290261062921262, "max": 13.290261062921262}}}
#metrics {"StartTime": 1655370520.6455963, "EndTime": 1655370520.6456602, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 13.421795414943322, "count": 1, "min": 13.421795414943322, "max": 13.421795414943322}}}
#metrics {"StartTime": 1655370520.6459756, "EndTime": 1655370520.64602, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.578905292585784, "count": 1, "min": 14.578905292585784, "max": 14.578905292585784}}}
#metrics {"StartTime": 1655370520.6463363, "EndTime": 1655370520.64638, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.51547660079657, "count": 1, "min": 14.51547660079657, "max": 14.51547660079657}}}
#metrics {"StartTime": 1655370520.6467245, "EndTime": 1655370520.64677, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.553095798866421, "count": 1, "min": 14.553095798866421, "max": 14.553095798866421}}}
#metrics {"StartTime": 1655370520.6470294, "EndTime": 1655370520.6470678, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.528086344401041, "count": 1, "min": 14.528086344401041, "max": 14.528086344401041}}}
#metrics {"StartTime": 1655370520.6473835, "EndTime": 1655370520.6474307, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 13.390285716337317, "count": 1, "min": 13.390285716337317, "max": 13.390285716337317}}}
#metrics {"StartTime": 1655370520.6477358, "EndTime": 1655370520.6477783, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.705990660424327, "count": 1, "min": 13.705990660424327, "max": 13.705990660424327}}}
#metrics {"StartTime": 1655370520.648081, "EndTime": 1655370520.6481264, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 13.814183253867954, "count": 1, "min": 13.814183253867954, "max": 13.814183253867954}}}
#metrics {"StartTime": 1655370520.6484518, "EndTime": 1655370520.6485014, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 13.45258705288756, "count": 1, "min": 13.45258705288756, "max": 13.45258705288756}}}
#metrics {"StartTime": 1655370520.6488178, "EndTime": 1655370520.6488643, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.407178691789216, "count": 1, "min": 14.407178691789216, "max": 14.407178691789216}}}
#metrics {"StartTime": 1655370520.649163, "EndTime": 1655370520.6492069, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 15.006446987974877, "count": 1, "min": 15.006446987974877, "max": 15.006446987974877}}}
#metrics {"StartTime": 1655370520.6495147, "EndTime": 1655370520.6495585, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.442462397556678, "count": 1, "min": 14.442462397556678, "max": 14.442462397556678}}}
#metrics {"StartTime": 1655370520.6498666, "EndTime": 1655370520.6499114, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.486004997702207, "count": 1, "min": 14.486004997702207, "max": 14.486004997702207}}}
#metrics {"StartTime": 1655370520.6502216, "EndTime": 1655370520.6502666, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.534964168772976, "count": 1, "min": 17.534964168772976, "max": 17.534964168772976}}}
#metrics {"StartTime": 1655370520.650586, "EndTime": 1655370520.6506255, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 17.617788277420342, "count": 1, "min": 17.617788277420342, "max": 17.617788277420342}}}
#metrics {"StartTime": 1655370520.650935, "EndTime": 1655370520.6509607, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.542332667930452, "count": 1, "min": 17.542332667930452, "max": 17.542332667930452}}}
#metrics {"StartTime": 1655370520.6512735, "EndTime": 1655370520.6513178, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.554816750919116, "count": 1, "min": 17.554816750919116, "max": 17.554816750919116}}}
#metrics {"StartTime": 1655370520.6516218, "EndTime": 1655370520.6516662, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.379646450865504, "count": 1, "min": 19.379646450865504, "max": 19.379646450865504}}}
#metrics {"StartTime": 1655370520.6519647, "EndTime": 1655370520.6520078, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.02353683172488, "count": 1, "min": 19.02353683172488, "max": 19.02353683172488}}}
#metrics {"StartTime": 1655370520.652338, "EndTime": 1655370520.6523826, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.319062476064644, "count": 1, "min": 19.319062476064644, "max": 19.319062476064644}}}
#metrics {"StartTime": 1655370520.6526477, "EndTime": 1655370520.652687, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.223100250842524, "count": 1, "min": 19.223100250842524, "max": 19.223100250842524}}}
#metrics {"StartTime": 1655370520.6530235, "EndTime": 1655370520.6530693, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.12922459022671, "count": 1, "min": 61.12922459022671, "max": 61.12922459022671}}}
#metrics {"StartTime": 1655370520.6533709, "EndTime": 1655370520.653412, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 60.997905656403184, "count": 1, "min": 60.997905656403184, "max": 60.997905656403184}}}
#metrics {"StartTime": 1655370520.6537628, "EndTime": 1655370520.6538067, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.44938869102329, "count": 1, "min": 61.44938869102329, "max": 61.44938869102329}}}
#metrics {"StartTime": 1655370520.6541333, "EndTime": 1655370520.6541798, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.21539665670956, "count": 1, "min": 61.21539665670956, "max": 61.21539665670956}}}
#metrics {"StartTime": 1655370520.6544425, "EndTime": 1655370520.6544812, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.395761527267155, "count": 1, "min": 61.395761527267155, "max": 61.395761527267155}}}
#metrics {"StartTime": 1655370520.6548033, "EndTime": 1655370520.65485, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 61.79503197763481, "count": 1, "min": 61.79503197763481, "max": 61.79503197763481}}}
#metrics {"StartTime": 1655370520.6551595, "EndTime": 1655370520.6552022, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 62.01028502221201, "count": 1, "min": 62.01028502221201, "max": 62.01028502221201}}}
#metrics {"StartTime": 1655370520.6555095, "EndTime": 1655370520.6555557, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.290199429381126, "count": 1, "min": 62.290199429381126, "max": 62.290199429381126}}}
[06/16/2022 09:08:40 INFO 139637139089216] #quality_metric: host=algo-1, epoch=9, validation mse_objective <loss>=13.868024040670957
[06/16/2022 09:08:40 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=9, criteria=mse_objective, value=13.290261062921262
[06/16/2022 09:08:40 INFO 139637139089216] Epoch 9: Loss has not improved for 0 epochs.
[06/16/2022 09:08:40 INFO 139637139089216] Saving model for epoch: 9
[06/16/2022 09:08:40 INFO 139637139089216] Saved checkpoint to "/tmp/tmpa9qtnet0/mx-mod-0000.params"
[06/16/2022 09:08:40 INFO 139637139089216] #progress_metric: host=algo-1, completed 66.66666666666667 % of epochs
#metrics {"StartTime": 1655370520.104647, "EndTime": 1655370520.6660225, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 9, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 5035.0, "count": 1, "min": 5035, "max": 5035}, "Total Batches Seen": {"sum": 177.0, "count": 1, "min": 177, "max": 177}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 12.0, "count": 1, "min": 12, "max": 12}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:40 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=809.7957188454478 records/second
#metrics {"StartTime": 1655370521.1553307, "EndTime": 1655370521.155433, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.2872430700725979, "count": 1, "min": 0.2872430700725979, "max": 0.2872430700725979}}}
#metrics {"StartTime": 1655370521.1560113, "EndTime": 1655370521.1560647, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.28935033798217774, "count": 1, "min": 0.28935033798217774, "max": 0.28935033798217774}}}
#metrics {"StartTime": 1655370521.1564155, "EndTime": 1655370521.1564596, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.28860703733232285, "count": 1, "min": 0.28860703733232285, "max": 0.28860703733232285}}}
#metrics {"StartTime": 1655370521.1567824, "EndTime": 1655370521.1568294, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.2859999407662286, "count": 1, "min": 0.2859999407662286, "max": 0.2859999407662286}}}
#metrics {"StartTime": 1655370521.1571589, "EndTime": 1655370521.1572099, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.30112812254163956, "count": 1, "min": 0.30112812254163956, "max": 0.30112812254163956}}}
#metrics {"StartTime": 1655370521.157527, "EndTime": 1655370521.157572, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.3003709740108914, "count": 1, "min": 0.3003709740108914, "max": 0.3003709740108914}}}
#metrics {"StartTime": 1655370521.1578062, "EndTime": 1655370521.1578603, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.29976721551683216, "count": 1, "min": 0.29976721551683216, "max": 0.29976721551683216}}}
#metrics {"StartTime": 1655370521.157921, "EndTime": 1655370521.1580422, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.3003331512875027, "count": 1, "min": 0.3003331512875027, "max": 0.3003331512875027}}}
#metrics {"StartTime": 1655370521.1580873, "EndTime": 1655370521.1580994, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.28624652438693576, "count": 1, "min": 0.28624652438693576, "max": 0.28624652438693576}}}
#metrics {"StartTime": 1655370521.1581507, "EndTime": 1655370521.1581624, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.2824632830089993, "count": 1, "min": 0.2824632830089993, "max": 0.2824632830089993}}}
#metrics {"StartTime": 1655370521.158344, "EndTime": 1655370521.1583612, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.2913515912161933, "count": 1, "min": 0.2913515912161933, "max": 0.2913515912161933}}}
#metrics {"StartTime": 1655370521.1584191, "EndTime": 1655370521.158431, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.2867222547531128, "count": 1, "min": 0.2867222547531128, "max": 0.2867222547531128}}}
#metrics {"StartTime": 1655370521.158595, "EndTime": 1655370521.1586115, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.2964517784118652, "count": 1, "min": 0.2964517784118652, "max": 0.2964517784118652}}}
#metrics {"StartTime": 1655370521.1586666, "EndTime": 1655370521.1586788, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.3138303894466824, "count": 1, "min": 0.3138303894466824, "max": 0.3138303894466824}}}
#metrics {"StartTime": 1655370521.158846, "EndTime": 1655370521.1588624, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.2951772732204861, "count": 1, "min": 0.2951772732204861, "max": 0.2951772732204861}}}
#metrics {"StartTime": 1655370521.158917, "EndTime": 1655370521.1589293, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.30056966675652397, "count": 1, "min": 0.30056966675652397, "max": 0.30056966675652397}}}
#metrics {"StartTime": 1655370521.158965, "EndTime": 1655370521.158974, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.37995175891452365, "count": 1, "min": 0.37995175891452365, "max": 0.37995175891452365}}}
#metrics {"StartTime": 1655370521.1590054, "EndTime": 1655370521.159015, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.38024831877814397, "count": 1, "min": 0.38024831877814397, "max": 0.38024831877814397}}}
#metrics {"StartTime": 1655370521.1590466, "EndTime": 1655370521.159056, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.3797888882954915, "count": 1, "min": 0.3797888882954915, "max": 0.3797888882954915}}}
#metrics {"StartTime": 1655370521.1590874, "EndTime": 1655370521.1590958, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.3799847994910346, "count": 1, "min": 0.3799847994910346, "max": 0.3799847994910346}}}
#metrics {"StartTime": 1655370521.1591284, "EndTime": 1655370521.159137, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.40136776606241864, "count": 1, "min": 0.40136776606241864, "max": 0.40136776606241864}}}
#metrics {"StartTime": 1655370521.1591702, "EndTime": 1655370521.1591794, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.41631723827785916, "count": 1, "min": 0.41631723827785916, "max": 0.41631723827785916}}}
#metrics {"StartTime": 1655370521.1595278, "EndTime": 1655370521.1595488, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4060533693101671, "count": 1, "min": 0.4060533693101671, "max": 0.4060533693101671}}}
#metrics {"StartTime": 1655370521.1596036, "EndTime": 1655370521.159616, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.40915524694654676, "count": 1, "min": 0.40915524694654676, "max": 0.40915524694654676}}}
#metrics {"StartTime": 1655370521.1596544, "EndTime": 1655370521.1596653, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9560007158915201, "count": 1, "min": 0.9560007158915201, "max": 0.9560007158915201}}}
#metrics {"StartTime": 1655370521.159742, "EndTime": 1655370521.1597555, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9564177046881782, "count": 1, "min": 0.9564177046881782, "max": 0.9564177046881782}}}
#metrics {"StartTime": 1655370521.159797, "EndTime": 1655370521.159842, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9556455760531956, "count": 1, "min": 0.9556455760531956, "max": 0.9556455760531956}}}
#metrics {"StartTime": 1655370521.1598837, "EndTime": 1655370521.1598947, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9562226380242241, "count": 1, "min": 0.9562226380242241, "max": 0.9562226380242241}}}
#metrics {"StartTime": 1655370521.159933, "EndTime": 1655370521.1599882, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9693977695041233, "count": 1, "min": 0.9693977695041233, "max": 0.9693977695041233}}}
#metrics {"StartTime": 1655370521.160031, "EndTime": 1655370521.1600428, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9774056010776095, "count": 1, "min": 0.9774056010776095, "max": 0.9774056010776095}}}
#metrics {"StartTime": 1655370521.1601214, "EndTime": 1655370521.160135, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9729545444912381, "count": 1, "min": 0.9729545444912381, "max": 0.9729545444912381}}}
#metrics {"StartTime": 1655370521.1601768, "EndTime": 1655370521.1601887, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9772756746080187, "count": 1, "min": 0.9772756746080187, "max": 0.9772756746080187}}}
[06/16/2022 09:08:41 INFO 139637139089216] #quality_metric: host=algo-1, epoch=10, train mse_objective <loss>=0.2872430700725979
#metrics {"StartTime": 1655370521.2678201, "EndTime": 1655370521.2679038, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 13.885411281211704, "count": 1, "min": 13.885411281211704, "max": 13.885411281211704}}}
#metrics {"StartTime": 1655370521.2680554, "EndTime": 1655370521.2680728, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 14.020633473115808, "count": 1, "min": 14.020633473115808, "max": 14.020633473115808}}}
#metrics {"StartTime": 1655370521.2681167, "EndTime": 1655370521.268127, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 13.289335961435356, "count": 1, "min": 13.289335961435356, "max": 13.289335961435356}}}
#metrics {"StartTime": 1655370521.2681606, "EndTime": 1655370521.2681696, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 13.498685948988971, "count": 1, "min": 13.498685948988971, "max": 13.498685948988971}}}
#metrics {"StartTime": 1655370521.2682428, "EndTime": 1655370521.268255, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.589147231158089, "count": 1, "min": 14.589147231158089, "max": 14.589147231158089}}}
#metrics {"StartTime": 1655370521.2682908, "EndTime": 1655370521.2682998, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.571391984528185, "count": 1, "min": 14.571391984528185, "max": 14.571391984528185}}}
#metrics {"StartTime": 1655370521.2683313, "EndTime": 1655370521.26834, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.57652671664369, "count": 1, "min": 14.57652671664369, "max": 14.57652671664369}}}
#metrics {"StartTime": 1655370521.2683728, "EndTime": 1655370521.2683816, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.593636307061887, "count": 1, "min": 14.593636307061887, "max": 14.593636307061887}}}
#metrics {"StartTime": 1655370521.2684186, "EndTime": 1655370521.268428, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 13.429443359375, "count": 1, "min": 13.429443359375, "max": 13.429443359375}}}
#metrics {"StartTime": 1655370521.2684615, "EndTime": 1655370521.2684712, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.770306157130822, "count": 1, "min": 13.770306157130822, "max": 13.770306157130822}}}
#metrics {"StartTime": 1655370521.2685473, "EndTime": 1655370521.2685602, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 13.774360207950368, "count": 1, "min": 13.774360207950368, "max": 13.774360207950368}}}
#metrics {"StartTime": 1655370521.268598, "EndTime": 1655370521.2686093, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 13.472411510991115, "count": 1, "min": 13.472411510991115, "max": 13.472411510991115}}}
#metrics {"StartTime": 1655370521.268646, "EndTime": 1655370521.2686553, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.376153684129902, "count": 1, "min": 14.376153684129902, "max": 14.376153684129902}}}
#metrics {"StartTime": 1655370521.26869, "EndTime": 1655370521.2687001, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 15.254106708601409, "count": 1, "min": 15.254106708601409, "max": 15.254106708601409}}}
#metrics {"StartTime": 1655370521.2687356, "EndTime": 1655370521.2687457, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.410895852481618, "count": 1, "min": 14.410895852481618, "max": 14.410895852481618}}}
#metrics {"StartTime": 1655370521.268781, "EndTime": 1655370521.2687902, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.556536506204043, "count": 1, "min": 14.556536506204043, "max": 14.556536506204043}}}
#metrics {"StartTime": 1655370521.2688277, "EndTime": 1655370521.268838, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.5369873046875, "count": 1, "min": 17.5369873046875, "max": 17.5369873046875}}}
#metrics {"StartTime": 1655370521.2688732, "EndTime": 1655370521.268883, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 17.582763073491115, "count": 1, "min": 17.582763073491115, "max": 17.582763073491115}}}
#metrics {"StartTime": 1655370521.2689197, "EndTime": 1655370521.2689302, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.53965130974265, "count": 1, "min": 17.53965130974265, "max": 17.53965130974265}}}
#metrics {"StartTime": 1655370521.2689676, "EndTime": 1655370521.2689779, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.54274076573989, "count": 1, "min": 17.54274076573989, "max": 17.54274076573989}}}
#metrics {"StartTime": 1655370521.2690144, "EndTime": 1655370521.2690246, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.389045266544116, "count": 1, "min": 19.389045266544116, "max": 19.389045266544116}}}
#metrics {"StartTime": 1655370521.269063, "EndTime": 1655370521.2690732, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.010003781786153, "count": 1, "min": 19.010003781786153, "max": 19.010003781786153}}}
#metrics {"StartTime": 1655370521.269109, "EndTime": 1655370521.2691188, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.310861026539524, "count": 1, "min": 19.310861026539524, "max": 19.310861026539524}}}
#metrics {"StartTime": 1655370521.2691545, "EndTime": 1655370521.2691648, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.197841868681067, "count": 1, "min": 19.197841868681067, "max": 19.197841868681067}}}
#metrics {"StartTime": 1655370521.2692018, "EndTime": 1655370521.2692115, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.11454743030024, "count": 1, "min": 61.11454743030024, "max": 61.11454743030024}}}
#metrics {"StartTime": 1655370521.2692473, "EndTime": 1655370521.2692573, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 60.98857326133579, "count": 1, "min": 60.98857326133579, "max": 60.98857326133579}}}
#metrics {"StartTime": 1655370521.2692924, "EndTime": 1655370521.2693026, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.43591069240196, "count": 1, "min": 61.43591069240196, "max": 61.43591069240196}}}
#metrics {"StartTime": 1655370521.269337, "EndTime": 1655370521.2693465, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.2046688304228, "count": 1, "min": 61.2046688304228, "max": 61.2046688304228}}}
#metrics {"StartTime": 1655370521.2693815, "EndTime": 1655370521.2693925, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.63593845741421, "count": 1, "min": 61.63593845741421, "max": 61.63593845741421}}}
#metrics {"StartTime": 1655370521.269429, "EndTime": 1655370521.269439, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 61.98420505897671, "count": 1, "min": 61.98420505897671, "max": 61.98420505897671}}}
#metrics {"StartTime": 1655370521.269476, "EndTime": 1655370521.2694857, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 62.02894263174019, "count": 1, "min": 62.02894263174019, "max": 62.02894263174019}}}
#metrics {"StartTime": 1655370521.2695224, "EndTime": 1655370521.2695951, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.67916331571691, "count": 1, "min": 62.67916331571691, "max": 62.67916331571691}}}
[06/16/2022 09:08:41 INFO 139637139089216] #quality_metric: host=algo-1, epoch=10, validation mse_objective <loss>=13.885411281211704
[06/16/2022 09:08:41 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=10, criteria=mse_objective, value=13.289335961435356
[06/16/2022 09:08:41 INFO 139637139089216] Saving model for epoch: 10
[06/16/2022 09:08:41 INFO 139637139089216] Saved checkpoint to "/tmp/tmpq9ykqpbv/mx-mod-0000.params"
[06/16/2022 09:08:41 INFO 139637139089216] #progress_metric: host=algo-1, completed 73.33333333333333 % of epochs
#metrics {"StartTime": 1655370520.6668208, "EndTime": 1655370521.279281, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 10, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 5490.0, "count": 1, "min": 5490, "max": 5490}, "Total Batches Seen": {"sum": 193.0, "count": 1, "min": 193, "max": 193}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 13.0, "count": 1, "min": 13, "max": 13}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:41 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=742.7218980968515 records/second
#metrics {"StartTime": 1655370521.838044, "EndTime": 1655370521.8381288, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.28510828071170385, "count": 1, "min": 0.28510828071170385, "max": 0.28510828071170385}}}
#metrics {"StartTime": 1655370521.8382256, "EndTime": 1655370521.83824, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.2867154428693983, "count": 1, "min": 0.2867154428693983, "max": 0.2867154428693983}}}
#metrics {"StartTime": 1655370521.8382823, "EndTime": 1655370521.8383014, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.28554254319932726, "count": 1, "min": 0.28554254319932726, "max": 0.28554254319932726}}}
#metrics {"StartTime": 1655370521.838337, "EndTime": 1655370521.8383467, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.2838771841261122, "count": 1, "min": 0.2838771841261122, "max": 0.2838771841261122}}}
#metrics {"StartTime": 1655370521.838379, "EndTime": 1655370521.8383877, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.3009424262576633, "count": 1, "min": 0.3009424262576633, "max": 0.3009424262576633}}}
#metrics {"StartTime": 1655370521.8384206, "EndTime": 1655370521.8384306, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.3021854559580485, "count": 1, "min": 0.3021854559580485, "max": 0.3021854559580485}}}
#metrics {"StartTime": 1655370521.8384652, "EndTime": 1655370521.838474, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.29982299592759876, "count": 1, "min": 0.29982299592759876, "max": 0.29982299592759876}}}
#metrics {"StartTime": 1655370521.8385057, "EndTime": 1655370521.8385148, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.3022594918145074, "count": 1, "min": 0.3022594918145074, "max": 0.3022594918145074}}}
#metrics {"StartTime": 1655370521.8385477, "EndTime": 1655370521.8385568, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.2839886787202623, "count": 1, "min": 0.2839886787202623, "max": 0.2839886787202623}}}
#metrics {"StartTime": 1655370521.8385937, "EndTime": 1655370521.8386028, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.28109286732143823, "count": 1, "min": 0.28109286732143823, "max": 0.28109286732143823}}}
#metrics {"StartTime": 1655370521.83868, "EndTime": 1655370521.8386939, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.2881507539749146, "count": 1, "min": 0.2881507539749146, "max": 0.2881507539749146}}}
#metrics {"StartTime": 1655370521.8387344, "EndTime": 1655370521.8387449, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.2841576279534234, "count": 1, "min": 0.2841576279534234, "max": 0.2841576279534234}}}
#metrics {"StartTime": 1655370521.8387816, "EndTime": 1655370521.8387928, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.2963763893975152, "count": 1, "min": 0.2963763893975152, "max": 0.2963763893975152}}}
#metrics {"StartTime": 1655370521.8388288, "EndTime": 1655370521.8388386, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.31561148643493653, "count": 1, "min": 0.31561148643493653, "max": 0.31561148643493653}}}
#metrics {"StartTime": 1655370521.8388734, "EndTime": 1655370521.8388822, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.29526398870680065, "count": 1, "min": 0.29526398870680065, "max": 0.29526398870680065}}}
#metrics {"StartTime": 1655370521.8389163, "EndTime": 1655370521.8389266, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.30245128101772734, "count": 1, "min": 0.30245128101772734, "max": 0.30245128101772734}}}
#metrics {"StartTime": 1655370521.8389623, "EndTime": 1655370521.8389711, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.3798140949673123, "count": 1, "min": 0.3798140949673123, "max": 0.3798140949673123}}}
#metrics {"StartTime": 1655370521.8390067, "EndTime": 1655370521.8390164, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.38002708541022406, "count": 1, "min": 0.38002708541022406, "max": 0.38002708541022406}}}
#metrics {"StartTime": 1655370521.8390517, "EndTime": 1655370521.839062, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.37976446893480087, "count": 1, "min": 0.37976446893480087, "max": 0.37976446893480087}}}
#metrics {"StartTime": 1655370521.8391478, "EndTime": 1655370521.8391607, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.3799731847974989, "count": 1, "min": 0.3799731847974989, "max": 0.3799731847974989}}}
#metrics {"StartTime": 1655370521.8391998, "EndTime": 1655370521.8392105, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.401339987648858, "count": 1, "min": 0.401339987648858, "max": 0.401339987648858}}}
#metrics {"StartTime": 1655370521.8392463, "EndTime": 1655370521.839256, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.4178485446506076, "count": 1, "min": 0.4178485446506076, "max": 0.4178485446506076}}}
#metrics {"StartTime": 1655370521.8392937, "EndTime": 1655370521.8393037, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.40586137029859753, "count": 1, "min": 0.40586137029859753, "max": 0.40586137029859753}}}
#metrics {"StartTime": 1655370521.8393395, "EndTime": 1655370521.8393495, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.4103736029730903, "count": 1, "min": 0.4103736029730903, "max": 0.4103736029730903}}}
#metrics {"StartTime": 1655370521.8393855, "EndTime": 1655370521.839395, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9560335731506348, "count": 1, "min": 0.9560335731506348, "max": 0.9560335731506348}}}
#metrics {"StartTime": 1655370521.839433, "EndTime": 1655370521.839443, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9564363055759006, "count": 1, "min": 0.9564363055759006, "max": 0.9564363055759006}}}
#metrics {"StartTime": 1655370521.8394783, "EndTime": 1655370521.8394887, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9556554137335883, "count": 1, "min": 0.9556554137335883, "max": 0.9556554137335883}}}
#metrics {"StartTime": 1655370521.8395252, "EndTime": 1655370521.8395355, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9562869029574924, "count": 1, "min": 0.9562869029574924, "max": 0.9562869029574924}}}
#metrics {"StartTime": 1655370521.8395717, "EndTime": 1655370521.8395817, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.969590695699056, "count": 1, "min": 0.969590695699056, "max": 0.969590695699056}}}
#metrics {"StartTime": 1655370521.8396208, "EndTime": 1655370521.839631, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9784575674268935, "count": 1, "min": 0.9784575674268935, "max": 0.9784575674268935}}}
#metrics {"StartTime": 1655370521.8396654, "EndTime": 1655370521.8396754, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9730250146653917, "count": 1, "min": 0.9730250146653917, "max": 0.9730250146653917}}}
#metrics {"StartTime": 1655370521.8397114, "EndTime": 1655370521.8397214, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.978685728708903, "count": 1, "min": 0.978685728708903, "max": 0.978685728708903}}}
[06/16/2022 09:08:41 INFO 139637139089216] #quality_metric: host=algo-1, epoch=11, train mse_objective <loss>=0.28510828071170385
#metrics {"StartTime": 1655370521.908292, "EndTime": 1655370521.908372, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 13.917296764897365, "count": 1, "min": 13.917296764897365, "max": 13.917296764897365}}}
#metrics {"StartTime": 1655370521.9084628, "EndTime": 1655370521.9084768, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 14.026091332529106, "count": 1, "min": 14.026091332529106, "max": 14.026091332529106}}}
#metrics {"StartTime": 1655370521.9085178, "EndTime": 1655370521.908528, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 13.328656963273591, "count": 1, "min": 13.328656963273591, "max": 13.328656963273591}}}
#metrics {"StartTime": 1655370521.9085612, "EndTime": 1655370521.9085703, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 13.589917949601716, "count": 1, "min": 13.589917949601716, "max": 13.589917949601716}}}
#metrics {"StartTime": 1655370521.9086041, "EndTime": 1655370521.9086134, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.57923500210631, "count": 1, "min": 14.57923500210631, "max": 14.57923500210631}}}
#metrics {"StartTime": 1655370521.9086478, "EndTime": 1655370521.9086573, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.614668303844976, "count": 1, "min": 14.614668303844976, "max": 14.614668303844976}}}
#metrics {"StartTime": 1655370521.9086905, "EndTime": 1655370521.9086993, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.565996357038909, "count": 1, "min": 14.565996357038909, "max": 14.565996357038909}}}
#metrics {"StartTime": 1655370521.9087322, "EndTime": 1655370521.9087415, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.629148595473346, "count": 1, "min": 14.629148595473346, "max": 14.629148595473346}}}
#metrics {"StartTime": 1655370521.9087734, "EndTime": 1655370521.9087832, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 13.480956432866115, "count": 1, "min": 13.480956432866115, "max": 13.480956432866115}}}
#metrics {"StartTime": 1655370521.9088173, "EndTime": 1655370521.9088268, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.820599724264707, "count": 1, "min": 13.820599724264707, "max": 13.820599724264707}}}
#metrics {"StartTime": 1655370521.9088614, "EndTime": 1655370521.9088717, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 13.776865042892156, "count": 1, "min": 13.776865042892156, "max": 13.776865042892156}}}
#metrics {"StartTime": 1655370521.9089086, "EndTime": 1655370521.9089189, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 13.53057741651348, "count": 1, "min": 13.53057741651348, "max": 13.53057741651348}}}
#metrics {"StartTime": 1655370521.9089546, "EndTime": 1655370521.9089653, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.385308957567402, "count": 1, "min": 14.385308957567402, "max": 14.385308957567402}}}
#metrics {"StartTime": 1655370521.909004, "EndTime": 1655370521.9090133, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 15.259306066176471, "count": 1, "min": 15.259306066176471, "max": 15.259306066176471}}}
#metrics {"StartTime": 1655370521.9090486, "EndTime": 1655370521.909058, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.419347426470589, "count": 1, "min": 14.419347426470589, "max": 14.419347426470589}}}
#metrics {"StartTime": 1655370521.9090927, "EndTime": 1655370521.909103, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.603858498965993, "count": 1, "min": 14.603858498965993, "max": 14.603858498965993}}}
#metrics {"StartTime": 1655370521.909139, "EndTime": 1655370521.909148, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.537115358838847, "count": 1, "min": 17.537115358838847, "max": 17.537115358838847}}}
#metrics {"StartTime": 1655370521.9091842, "EndTime": 1655370521.9091947, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 17.564494413488053, "count": 1, "min": 17.564494413488053, "max": 17.564494413488053}}}
#metrics {"StartTime": 1655370521.9092293, "EndTime": 1655370521.9092398, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.540641635071996, "count": 1, "min": 17.540641635071996, "max": 17.540641635071996}}}
#metrics {"StartTime": 1655370521.9092762, "EndTime": 1655370521.9092863, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.54013659907322, "count": 1, "min": 17.54013659907322, "max": 17.54013659907322}}}
#metrics {"StartTime": 1655370521.9093225, "EndTime": 1655370521.9093328, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.40724271886489, "count": 1, "min": 19.40724271886489, "max": 19.40724271886489}}}
#metrics {"StartTime": 1655370521.909369, "EndTime": 1655370521.9093795, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.024035883884803, "count": 1, "min": 19.024035883884803, "max": 19.024035883884803}}}
#metrics {"StartTime": 1655370521.9094188, "EndTime": 1655370521.9094286, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.3359375, "count": 1, "min": 19.3359375, "max": 19.3359375}}}
#metrics {"StartTime": 1655370521.9094682, "EndTime": 1655370521.9094777, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.152896656709558, "count": 1, "min": 19.152896656709558, "max": 19.152896656709558}}}
#metrics {"StartTime": 1655370521.9095163, "EndTime": 1655370521.909527, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.09646187576593, "count": 1, "min": 61.09646187576593, "max": 61.09646187576593}}}
#metrics {"StartTime": 1655370521.909563, "EndTime": 1655370521.9095724, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 60.98802035462622, "count": 1, "min": 60.98802035462622, "max": 60.98802035462622}}}
#metrics {"StartTime": 1655370521.909608, "EndTime": 1655370521.9096174, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.4263365502451, "count": 1, "min": 61.4263365502451, "max": 61.4263365502451}}}
#metrics {"StartTime": 1655370521.9096725, "EndTime": 1655370521.9096832, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.19737572763481, "count": 1, "min": 61.19737572763481, "max": 61.19737572763481}}}
#metrics {"StartTime": 1655370521.9097197, "EndTime": 1655370521.909729, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.81381644454657, "count": 1, "min": 61.81381644454657, "max": 61.81381644454657}}}
#metrics {"StartTime": 1655370521.909764, "EndTime": 1655370521.9097743, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 62.171288583792894, "count": 1, "min": 62.171288583792894, "max": 62.171288583792894}}}
#metrics {"StartTime": 1655370521.9098115, "EndTime": 1655370521.9098217, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 61.923943014705884, "count": 1, "min": 61.923943014705884, "max": 61.923943014705884}}}
#metrics {"StartTime": 1655370521.9098577, "EndTime": 1655370521.909868, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.848561006433826, "count": 1, "min": 62.848561006433826, "max": 62.848561006433826}}}
[06/16/2022 09:08:41 INFO 139637139089216] #quality_metric: host=algo-1, epoch=11, validation mse_objective <loss>=13.917296764897365
[06/16/2022 09:08:41 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=11, criteria=mse_objective, value=13.328656963273591
[06/16/2022 09:08:41 INFO 139637139089216] Saving model for epoch: 11
[06/16/2022 09:08:41 INFO 139637139089216] Saved checkpoint to "/tmp/tmpvut95rhe/mx-mod-0000.params"
[06/16/2022 09:08:41 INFO 139637139089216] #progress_metric: host=algo-1, completed 80.0 % of epochs
#metrics {"StartTime": 1655370521.2795324, "EndTime": 1655370521.933144, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 11, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 5945.0, "count": 1, "min": 5945, "max": 5945}, "Total Batches Seen": {"sum": 209.0, "count": 1, "min": 209, "max": 209}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 14.0, "count": 1, "min": 14, "max": 14}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:41 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=695.6503887226684 records/second
#metrics {"StartTime": 1655370522.744168, "EndTime": 1655370522.744255, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.28346955882178415, "count": 1, "min": 0.28346955882178415, "max": 0.28346955882178415}}}
#metrics {"StartTime": 1655370522.7448397, "EndTime": 1655370522.7448685, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.28467477321624757, "count": 1, "min": 0.28467477321624757, "max": 0.28467477321624757}}}
#metrics {"StartTime": 1655370522.7452223, "EndTime": 1655370522.7452695, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.2833048539691501, "count": 1, "min": 0.2833048539691501, "max": 0.2833048539691501}}}
#metrics {"StartTime": 1655370522.745621, "EndTime": 1655370522.7456656, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.28234869215223524, "count": 1, "min": 0.28234869215223524, "max": 0.28234869215223524}}}
#metrics {"StartTime": 1655370522.7460055, "EndTime": 1655370522.7460508, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.3009205606248644, "count": 1, "min": 0.3009205606248644, "max": 0.3009205606248644}}}
#metrics {"StartTime": 1655370522.7463856, "EndTime": 1655370522.7464309, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.3041692638397217, "count": 1, "min": 0.3041692638397217, "max": 0.3041692638397217}}}
#metrics {"StartTime": 1655370522.746754, "EndTime": 1655370522.7467983, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.2999672328101264, "count": 1, "min": 0.2999672328101264, "max": 0.2999672328101264}}}
#metrics {"StartTime": 1655370522.7471278, "EndTime": 1655370522.7471726, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.30425420337253145, "count": 1, "min": 0.30425420337253145, "max": 0.30425420337253145}}}
#metrics {"StartTime": 1655370522.747513, "EndTime": 1655370522.7475533, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.28228312651316323, "count": 1, "min": 0.28228312651316323, "max": 0.28228312651316323}}}
#metrics {"StartTime": 1655370522.7478898, "EndTime": 1655370522.7479336, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.2800179158316718, "count": 1, "min": 0.2800179158316718, "max": 0.2800179158316718}}}
#metrics {"StartTime": 1655370522.7482476, "EndTime": 1655370522.7482934, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.2857889880074395, "count": 1, "min": 0.2857889880074395, "max": 0.2857889880074395}}}
#metrics {"StartTime": 1655370522.7486112, "EndTime": 1655370522.7486553, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.28236386722988555, "count": 1, "min": 0.28236386722988555, "max": 0.28236386722988555}}}
#metrics {"StartTime": 1655370522.748978, "EndTime": 1655370522.7490282, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.2964678043789334, "count": 1, "min": 0.2964678043789334, "max": 0.2964678043789334}}}
#metrics {"StartTime": 1655370522.7493563, "EndTime": 1655370522.7494028, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.31731542693244086, "count": 1, "min": 0.31731542693244086, "max": 0.31731542693244086}}}
#metrics {"StartTime": 1655370522.7497418, "EndTime": 1655370522.7497861, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.2955373117658827, "count": 1, "min": 0.2955373117658827, "max": 0.2955373117658827}}}
#metrics {"StartTime": 1655370522.7501106, "EndTime": 1655370522.7501576, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.30444920751783583, "count": 1, "min": 0.30444920751783583, "max": 0.30444920751783583}}}
#metrics {"StartTime": 1655370522.750486, "EndTime": 1655370522.750531, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.3797358269161648, "count": 1, "min": 0.3797358269161648, "max": 0.3797358269161648}}}
#metrics {"StartTime": 1655370522.7508702, "EndTime": 1655370522.7509215, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.3799650086296929, "count": 1, "min": 0.3799650086296929, "max": 0.3799650086296929}}}
#metrics {"StartTime": 1655370522.7512617, "EndTime": 1655370522.7513063, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.3797464413113064, "count": 1, "min": 0.3797464413113064, "max": 0.3797464413113064}}}
#metrics {"StartTime": 1655370522.7516298, "EndTime": 1655370522.7516758, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.3799987326727973, "count": 1, "min": 0.3799987326727973, "max": 0.3799987326727973}}}
#metrics {"StartTime": 1655370522.7519953, "EndTime": 1655370522.7520382, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.40152807659573025, "count": 1, "min": 0.40152807659573025, "max": 0.40152807659573025}}}
#metrics {"StartTime": 1655370522.752357, "EndTime": 1655370522.752402, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.4192352294921875, "count": 1, "min": 0.4192352294921875, "max": 0.4192352294921875}}}
#metrics {"StartTime": 1655370522.7527537, "EndTime": 1655370522.7527943, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4058685482872857, "count": 1, "min": 0.4058685482872857, "max": 0.4058685482872857}}}
#metrics {"StartTime": 1655370522.7531254, "EndTime": 1655370522.7531717, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.41155166731940374, "count": 1, "min": 0.41155166731940374, "max": 0.41155166731940374}}}
#metrics {"StartTime": 1655370522.7535152, "EndTime": 1655370522.7535622, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9560334947374132, "count": 1, "min": 0.9560334947374132, "max": 0.9560334947374132}}}
#metrics {"StartTime": 1655370522.7538936, "EndTime": 1655370522.753939, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9564646212259929, "count": 1, "min": 0.9564646212259929, "max": 0.9564646212259929}}}
#metrics {"StartTime": 1655370522.7542644, "EndTime": 1655370522.754341, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9556433169047037, "count": 1, "min": 0.9556433169047037, "max": 0.9556433169047037}}}
#metrics {"StartTime": 1655370522.7547088, "EndTime": 1655370522.754759, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.9563207986619737, "count": 1, "min": 0.9563207986619737, "max": 0.9563207986619737}}}
#metrics {"StartTime": 1655370522.7550771, "EndTime": 1655370522.7551222, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9697733688354492, "count": 1, "min": 0.9697733688354492, "max": 0.9697733688354492}}}
#metrics {"StartTime": 1655370522.7554772, "EndTime": 1655370522.755521, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9795195134480794, "count": 1, "min": 0.9795195134480794, "max": 0.9795195134480794}}}
#metrics {"StartTime": 1655370522.75584, "EndTime": 1655370522.7558835, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9728144900004069, "count": 1, "min": 0.9728144900004069, "max": 0.9728144900004069}}}
#metrics {"StartTime": 1655370522.7562275, "EndTime": 1655370522.75627, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9797180620829264, "count": 1, "min": 0.9797180620829264, "max": 0.9797180620829264}}}
[06/16/2022 09:08:42 INFO 139637139089216] #quality_metric: host=algo-1, epoch=12, train mse_objective <loss>=0.28346955882178415
#metrics {"StartTime": 1655370522.8490186, "EndTime": 1655370522.8491027, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 13.952870686848959, "count": 1, "min": 13.952870686848959, "max": 13.952870686848959}}}
#metrics {"StartTime": 1655370522.8497438, "EndTime": 1655370522.8497763, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 14.041442272709865, "count": 1, "min": 14.041442272709865, "max": 14.041442272709865}}}
#metrics {"StartTime": 1655370522.8501592, "EndTime": 1655370522.850207, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 13.387701296338848, "count": 1, "min": 13.387701296338848, "max": 13.387701296338848}}}
#metrics {"StartTime": 1655370522.850549, "EndTime": 1655370522.850595, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 13.677093505859375, "count": 1, "min": 13.677093505859375, "max": 13.677093505859375}}}
#metrics {"StartTime": 1655370522.8509276, "EndTime": 1655370522.8509762, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.571622362323836, "count": 1, "min": 14.571622362323836, "max": 14.571622362323836}}}
#metrics {"StartTime": 1655370522.851315, "EndTime": 1655370522.8513618, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.684067670036764, "count": 1, "min": 14.684067670036764, "max": 14.684067670036764}}}
#metrics {"StartTime": 1655370522.8516748, "EndTime": 1655370522.8517213, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.562893138212317, "count": 1, "min": 14.562893138212317, "max": 14.562893138212317}}}
#metrics {"StartTime": 1655370522.8520532, "EndTime": 1655370522.8520968, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.698338527305454, "count": 1, "min": 14.698338527305454, "max": 14.698338527305454}}}
#metrics {"StartTime": 1655370522.8524196, "EndTime": 1655370522.8524632, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 13.534875607958027, "count": 1, "min": 13.534875607958027, "max": 13.534875607958027}}}
#metrics {"StartTime": 1655370522.852819, "EndTime": 1655370522.8528612, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.85780424230239, "count": 1, "min": 13.85780424230239, "max": 13.85780424230239}}}
#metrics {"StartTime": 1655370522.853187, "EndTime": 1655370522.8532326, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 13.80215035232843, "count": 1, "min": 13.80215035232843, "max": 13.80215035232843}}}
#metrics {"StartTime": 1655370522.8535538, "EndTime": 1655370522.8535995, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 13.598674699371935, "count": 1, "min": 13.598674699371935, "max": 13.598674699371935}}}
#metrics {"StartTime": 1655370522.8544183, "EndTime": 1655370522.8544478, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.3717788995481, "count": 1, "min": 14.3717788995481, "max": 14.3717788995481}}}
#metrics {"StartTime": 1655370522.8547835, "EndTime": 1655370522.8548334, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 15.38017661898744, "count": 1, "min": 15.38017661898744, "max": 15.38017661898744}}}
#metrics {"StartTime": 1655370522.855156, "EndTime": 1655370522.855203, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.405591179342832, "count": 1, "min": 14.405591179342832, "max": 14.405591179342832}}}
#metrics {"StartTime": 1655370522.8555677, "EndTime": 1655370522.8556118, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.681273815678615, "count": 1, "min": 14.681273815678615, "max": 14.681273815678615}}}
#metrics {"StartTime": 1655370522.8559337, "EndTime": 1655370522.8559775, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.541234633501837, "count": 1, "min": 17.541234633501837, "max": 17.541234633501837}}}
#metrics {"StartTime": 1655370522.8563218, "EndTime": 1655370522.8563619, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 17.555366067325366, "count": 1, "min": 17.555366067325366, "max": 17.555366067325366}}}
#metrics {"StartTime": 1655370522.856702, "EndTime": 1655370522.8567526, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.542622285730697, "count": 1, "min": 17.542622285730697, "max": 17.542622285730697}}}
#metrics {"StartTime": 1655370522.8570683, "EndTime": 1655370522.857115, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.537333170572918, "count": 1, "min": 17.537333170572918, "max": 17.537333170572918}}}
#metrics {"StartTime": 1655370522.8574288, "EndTime": 1655370522.8574727, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.39291082643995, "count": 1, "min": 19.39291082643995, "max": 19.39291082643995}}}
#metrics {"StartTime": 1655370522.85785, "EndTime": 1655370522.8579016, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.05091588637408, "count": 1, "min": 19.05091588637408, "max": 19.05091588637408}}}
#metrics {"StartTime": 1655370522.8582387, "EndTime": 1655370522.8582804, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.31894519282322, "count": 1, "min": 19.31894519282322, "max": 19.31894519282322}}}
#metrics {"StartTime": 1655370522.8586054, "EndTime": 1655370522.8586512, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.117049871706495, "count": 1, "min": 19.117049871706495, "max": 19.117049871706495}}}
#metrics {"StartTime": 1655370522.859465, "EndTime": 1655370522.8595097, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.08127249923407, "count": 1, "min": 61.08127249923407, "max": 61.08127249923407}}}
#metrics {"StartTime": 1655370522.8598611, "EndTime": 1655370522.8599038, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 60.99089259727329, "count": 1, "min": 60.99089259727329, "max": 60.99089259727329}}}
#metrics {"StartTime": 1655370522.860268, "EndTime": 1655370522.8603106, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.41563744638481, "count": 1, "min": 61.41563744638481, "max": 61.41563744638481}}}
#metrics {"StartTime": 1655370522.860659, "EndTime": 1655370522.8607047, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.1947691674326, "count": 1, "min": 61.1947691674326, "max": 61.1947691674326}}}
#metrics {"StartTime": 1655370522.8610227, "EndTime": 1655370522.8610675, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.909871419270836, "count": 1, "min": 61.909871419270836, "max": 61.909871419270836}}}
#metrics {"StartTime": 1655370522.8613935, "EndTime": 1655370522.8614388, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 62.42349063648897, "count": 1, "min": 62.42349063648897, "max": 62.42349063648897}}}
#metrics {"StartTime": 1655370522.8617764, "EndTime": 1655370522.8618243, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 61.9844970703125, "count": 1, "min": 61.9844970703125, "max": 61.9844970703125}}}
#metrics {"StartTime": 1655370522.862132, "EndTime": 1655370522.862174, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.90167356004902, "count": 1, "min": 62.90167356004902, "max": 62.90167356004902}}}
[06/16/2022 09:08:42 INFO 139637139089216] #quality_metric: host=algo-1, epoch=12, validation mse_objective <loss>=13.952870686848959
[06/16/2022 09:08:42 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=12, criteria=mse_objective, value=13.387701296338848
[06/16/2022 09:08:42 INFO 139637139089216] Saving model for epoch: 12
[06/16/2022 09:08:42 INFO 139637139089216] Saved checkpoint to "/tmp/tmp34kxp_1v/mx-mod-0000.params"
[06/16/2022 09:08:42 INFO 139637139089216] #progress_metric: host=algo-1, completed 86.66666666666667 % of epochs
#metrics {"StartTime": 1655370521.9340303, "EndTime": 1655370522.8723795, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 12, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 6400.0, "count": 1, "min": 6400, "max": 6400}, "Total Batches Seen": {"sum": 225.0, "count": 1, "min": 225, "max": 225}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 15.0, "count": 1, "min": 15, "max": 15}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:42 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=484.65986594961834 records/second
#metrics {"StartTime": 1655370523.309051, "EndTime": 1655370523.3091316, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 0}, "Metrics": {"train_mse_objective": {"sum": 0.2821601152420044, "count": 1, "min": 0.2821601152420044, "max": 0.2821601152420044}}}
#metrics {"StartTime": 1655370523.3092227, "EndTime": 1655370523.3095124, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 1}, "Metrics": {"train_mse_objective": {"sum": 0.28304072274102104, "count": 1, "min": 0.28304072274102104, "max": 0.28304072274102104}}}
#metrics {"StartTime": 1655370523.3096056, "EndTime": 1655370523.3098087, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 2}, "Metrics": {"train_mse_objective": {"sum": 0.281617022090488, "count": 1, "min": 0.281617022090488, "max": 0.281617022090488}}}
#metrics {"StartTime": 1655370523.3098888, "EndTime": 1655370523.3099086, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 3}, "Metrics": {"train_mse_objective": {"sum": 0.28118344730801054, "count": 1, "min": 0.28118344730801054, "max": 0.28118344730801054}}}
#metrics {"StartTime": 1655370523.3099592, "EndTime": 1655370523.3099697, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 4}, "Metrics": {"train_mse_objective": {"sum": 0.300567405488756, "count": 1, "min": 0.300567405488756, "max": 0.300567405488756}}}
#metrics {"StartTime": 1655370523.3100107, "EndTime": 1655370523.3100214, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 5}, "Metrics": {"train_mse_objective": {"sum": 0.3058248339758979, "count": 1, "min": 0.3058248339758979, "max": 0.3058248339758979}}}
#metrics {"StartTime": 1655370523.3100584, "EndTime": 1655370523.3100681, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 6}, "Metrics": {"train_mse_objective": {"sum": 0.2997693220774333, "count": 1, "min": 0.2997693220774333, "max": 0.2997693220774333}}}
#metrics {"StartTime": 1655370523.310537, "EndTime": 1655370523.3105624, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 7}, "Metrics": {"train_mse_objective": {"sum": 0.3058673487769233, "count": 1, "min": 0.3058673487769233, "max": 0.3058673487769233}}}
#metrics {"StartTime": 1655370523.3107874, "EndTime": 1655370523.3108113, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 8}, "Metrics": {"train_mse_objective": {"sum": 0.2809463940726386, "count": 1, "min": 0.2809463940726386, "max": 0.2809463940726386}}}
#metrics {"StartTime": 1655370523.3108766, "EndTime": 1655370523.3110325, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 9}, "Metrics": {"train_mse_objective": {"sum": 0.2791322316063775, "count": 1, "min": 0.2791322316063775, "max": 0.2791322316063775}}}
#metrics {"StartTime": 1655370523.311095, "EndTime": 1655370523.3112502, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 10}, "Metrics": {"train_mse_objective": {"sum": 0.2839924584494697, "count": 1, "min": 0.2839924584494697, "max": 0.2839924584494697}}}
#metrics {"StartTime": 1655370523.311314, "EndTime": 1655370523.3115017, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 11}, "Metrics": {"train_mse_objective": {"sum": 0.28104520426856144, "count": 1, "min": 0.28104520426856144, "max": 0.28104520426856144}}}
#metrics {"StartTime": 1655370523.3115702, "EndTime": 1655370523.3117313, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 12}, "Metrics": {"train_mse_objective": {"sum": 0.2964671399858263, "count": 1, "min": 0.2964671399858263, "max": 0.2964671399858263}}}
#metrics {"StartTime": 1655370523.3117957, "EndTime": 1655370523.3119678, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 13}, "Metrics": {"train_mse_objective": {"sum": 0.3177454800075955, "count": 1, "min": 0.3177454800075955, "max": 0.3177454800075955}}}
#metrics {"StartTime": 1655370523.3121834, "EndTime": 1655370523.3122082, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 14}, "Metrics": {"train_mse_objective": {"sum": 0.2956636407640245, "count": 1, "min": 0.2956636407640245, "max": 0.2956636407640245}}}
#metrics {"StartTime": 1655370523.3122656, "EndTime": 1655370523.3124177, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 15}, "Metrics": {"train_mse_objective": {"sum": 0.30609261088901096, "count": 1, "min": 0.30609261088901096, "max": 0.30609261088901096}}}
#metrics {"StartTime": 1655370523.3124983, "EndTime": 1655370523.3125155, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 16}, "Metrics": {"train_mse_objective": {"sum": 0.37971383624606664, "count": 1, "min": 0.37971383624606664, "max": 0.37971383624606664}}}
#metrics {"StartTime": 1655370523.3125606, "EndTime": 1655370523.3125713, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 17}, "Metrics": {"train_mse_objective": {"sum": 0.379945953157213, "count": 1, "min": 0.379945953157213, "max": 0.379945953157213}}}
#metrics {"StartTime": 1655370523.3126125, "EndTime": 1655370523.3126242, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 18}, "Metrics": {"train_mse_objective": {"sum": 0.3797486570146349, "count": 1, "min": 0.3797486570146349, "max": 0.3797486570146349}}}
#metrics {"StartTime": 1655370523.3126614, "EndTime": 1655370523.312672, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 19}, "Metrics": {"train_mse_objective": {"sum": 0.3800104278988308, "count": 1, "min": 0.3800104278988308, "max": 0.3800104278988308}}}
#metrics {"StartTime": 1655370523.3129928, "EndTime": 1655370523.313109, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 20}, "Metrics": {"train_mse_objective": {"sum": 0.40137112193637425, "count": 1, "min": 0.40137112193637425, "max": 0.40137112193637425}}}
#metrics {"StartTime": 1655370523.3131723, "EndTime": 1655370523.3131878, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 21}, "Metrics": {"train_mse_objective": {"sum": 0.4200400214725071, "count": 1, "min": 0.4200400214725071, "max": 0.4200400214725071}}}
#metrics {"StartTime": 1655370523.3135312, "EndTime": 1655370523.3135555, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 22}, "Metrics": {"train_mse_objective": {"sum": 0.4054637707604302, "count": 1, "min": 0.4054637707604302, "max": 0.4054637707604302}}}
#metrics {"StartTime": 1655370523.313618, "EndTime": 1655370523.3137882, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 23}, "Metrics": {"train_mse_objective": {"sum": 0.4123759068383111, "count": 1, "min": 0.4123759068383111, "max": 0.4123759068383111}}}
#metrics {"StartTime": 1655370523.3138504, "EndTime": 1655370523.314, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 24}, "Metrics": {"train_mse_objective": {"sum": 0.9560433302985297, "count": 1, "min": 0.9560433302985297, "max": 0.9560433302985297}}}
#metrics {"StartTime": 1655370523.3140628, "EndTime": 1655370523.3142183, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 25}, "Metrics": {"train_mse_objective": {"sum": 0.9565350829230415, "count": 1, "min": 0.9565350829230415, "max": 0.9565350829230415}}}
#metrics {"StartTime": 1655370523.3144197, "EndTime": 1655370523.314445, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 26}, "Metrics": {"train_mse_objective": {"sum": 0.9556538920932346, "count": 1, "min": 0.9556538920932346, "max": 0.9556538920932346}}}
#metrics {"StartTime": 1655370523.3145072, "EndTime": 1655370523.3146608, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 27}, "Metrics": {"train_mse_objective": {"sum": 0.956398237016466, "count": 1, "min": 0.956398237016466, "max": 0.956398237016466}}}
#metrics {"StartTime": 1655370523.3147306, "EndTime": 1655370523.3148854, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 28}, "Metrics": {"train_mse_objective": {"sum": 0.9698759672376844, "count": 1, "min": 0.9698759672376844, "max": 0.9698759672376844}}}
#metrics {"StartTime": 1655370523.314955, "EndTime": 1655370523.315116, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 29}, "Metrics": {"train_mse_objective": {"sum": 0.9803103680080838, "count": 1, "min": 0.9803103680080838, "max": 0.9803103680080838}}}
#metrics {"StartTime": 1655370523.3151734, "EndTime": 1655370523.315281, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 30}, "Metrics": {"train_mse_objective": {"sum": 0.9728766738043891, "count": 1, "min": 0.9728766738043891, "max": 0.9728766738043891}}}
#metrics {"StartTime": 1655370523.3153994, "EndTime": 1655370523.3154194, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 31}, "Metrics": {"train_mse_objective": {"sum": 0.9802551205952962, "count": 1, "min": 0.9802551205952962, "max": 0.9802551205952962}}}
[06/16/2022 09:08:43 INFO 139637139089216] #quality_metric: host=algo-1, epoch=13, train mse_objective <loss>=0.2821601152420044
#metrics {"StartTime": 1655370523.3924441, "EndTime": 1655370523.3924918, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 0}, "Metrics": {"validation_mse_objective": {"sum": 13.984094357958027, "count": 1, "min": 13.984094357958027, "max": 13.984094357958027}}}
#metrics {"StartTime": 1655370523.392559, "EndTime": 1655370523.3928542, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 1}, "Metrics": {"validation_mse_objective": {"sum": 14.058059991574755, "count": 1, "min": 14.058059991574755, "max": 14.058059991574755}}}
#metrics {"StartTime": 1655370523.3929365, "EndTime": 1655370523.3929565, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 2}, "Metrics": {"validation_mse_objective": {"sum": 13.450200697954964, "count": 1, "min": 13.450200697954964, "max": 13.450200697954964}}}
#metrics {"StartTime": 1655370523.3931112, "EndTime": 1655370523.3931284, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 3}, "Metrics": {"validation_mse_objective": {"sum": 13.753068512561274, "count": 1, "min": 13.753068512561274, "max": 13.753068512561274}}}
#metrics {"StartTime": 1655370523.3931706, "EndTime": 1655370523.3931825, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 4}, "Metrics": {"validation_mse_objective": {"sum": 14.57038490445006, "count": 1, "min": 14.57038490445006, "max": 14.57038490445006}}}
#metrics {"StartTime": 1655370523.3932192, "EndTime": 1655370523.3932292, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 5}, "Metrics": {"validation_mse_objective": {"sum": 14.710671219171262, "count": 1, "min": 14.710671219171262, "max": 14.710671219171262}}}
#metrics {"StartTime": 1655370523.3934238, "EndTime": 1655370523.393508, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 6}, "Metrics": {"validation_mse_objective": {"sum": 14.560327866498161, "count": 1, "min": 14.560327866498161, "max": 14.560327866498161}}}
#metrics {"StartTime": 1655370523.3935683, "EndTime": 1655370523.3936737, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 7}, "Metrics": {"validation_mse_objective": {"sum": 14.722483317057291, "count": 1, "min": 14.722483317057291, "max": 14.722483317057291}}}
#metrics {"StartTime": 1655370523.3937376, "EndTime": 1655370523.3938339, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 8}, "Metrics": {"validation_mse_objective": {"sum": 13.583147235945159, "count": 1, "min": 13.583147235945159, "max": 13.583147235945159}}}
#metrics {"StartTime": 1655370523.3938925, "EndTime": 1655370523.393978, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 9}, "Metrics": {"validation_mse_objective": {"sum": 13.884679457720589, "count": 1, "min": 13.884679457720589, "max": 13.884679457720589}}}
#metrics {"StartTime": 1655370523.3940337, "EndTime": 1655370523.3941226, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 10}, "Metrics": {"validation_mse_objective": {"sum": 13.834466672411152, "count": 1, "min": 13.834466672411152, "max": 13.834466672411152}}}
#metrics {"StartTime": 1655370523.3941772, "EndTime": 1655370523.3942597, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 11}, "Metrics": {"validation_mse_objective": {"sum": 13.663813572303921, "count": 1, "min": 13.663813572303921, "max": 13.663813572303921}}}
#metrics {"StartTime": 1655370523.39432, "EndTime": 1655370523.3944044, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 12}, "Metrics": {"validation_mse_objective": {"sum": 14.375144808900123, "count": 1, "min": 14.375144808900123, "max": 14.375144808900123}}}
#metrics {"StartTime": 1655370523.3944583, "EndTime": 1655370523.3945484, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 13}, "Metrics": {"validation_mse_objective": {"sum": 15.29500923904718, "count": 1, "min": 15.29500923904718, "max": 15.29500923904718}}}
#metrics {"StartTime": 1655370523.394603, "EndTime": 1655370523.3946898, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 14}, "Metrics": {"validation_mse_objective": {"sum": 14.410013236251531, "count": 1, "min": 14.410013236251531, "max": 14.410013236251531}}}
#metrics {"StartTime": 1655370523.3947444, "EndTime": 1655370523.3948307, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 15}, "Metrics": {"validation_mse_objective": {"sum": 14.709462483723959, "count": 1, "min": 14.709462483723959, "max": 14.709462483723959}}}
#metrics {"StartTime": 1655370523.3948846, "EndTime": 1655370523.3949654, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 16}, "Metrics": {"validation_mse_objective": {"sum": 17.542254279641543, "count": 1, "min": 17.542254279641543, "max": 17.542254279641543}}}
#metrics {"StartTime": 1655370523.3950195, "EndTime": 1655370523.3951025, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 17}, "Metrics": {"validation_mse_objective": {"sum": 17.55028638652727, "count": 1, "min": 17.55028638652727, "max": 17.55028638652727}}}
#metrics {"StartTime": 1655370523.3951552, "EndTime": 1655370523.395237, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 18}, "Metrics": {"validation_mse_objective": {"sum": 17.541951497395832, "count": 1, "min": 17.541951497395832, "max": 17.541951497395832}}}
#metrics {"StartTime": 1655370523.3952894, "EndTime": 1655370523.3953753, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 19}, "Metrics": {"validation_mse_objective": {"sum": 17.536605535768995, "count": 1, "min": 17.536605535768995, "max": 17.536605535768995}}}
#metrics {"StartTime": 1655370523.3954294, "EndTime": 1655370523.3955135, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 20}, "Metrics": {"validation_mse_objective": {"sum": 19.408946317784928, "count": 1, "min": 19.408946317784928, "max": 19.408946317784928}}}
#metrics {"StartTime": 1655370523.3955681, "EndTime": 1655370523.3956475, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 21}, "Metrics": {"validation_mse_objective": {"sum": 19.091358259612438, "count": 1, "min": 19.091358259612438, "max": 19.091358259612438}}}
#metrics {"StartTime": 1655370523.3956995, "EndTime": 1655370523.3957841, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 22}, "Metrics": {"validation_mse_objective": {"sum": 19.353264303768384, "count": 1, "min": 19.353264303768384, "max": 19.353264303768384}}}
#metrics {"StartTime": 1655370523.3958383, "EndTime": 1655370523.395919, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 23}, "Metrics": {"validation_mse_objective": {"sum": 19.1146635167739, "count": 1, "min": 19.1146635167739, "max": 19.1146635167739}}}
#metrics {"StartTime": 1655370523.3959725, "EndTime": 1655370523.3960545, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 24}, "Metrics": {"validation_mse_objective": {"sum": 61.06577914368872, "count": 1, "min": 61.06577914368872, "max": 61.06577914368872}}}
#metrics {"StartTime": 1655370523.3961074, "EndTime": 1655370523.3962035, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 25}, "Metrics": {"validation_mse_objective": {"sum": 60.98327158011642, "count": 1, "min": 60.98327158011642, "max": 60.98327158011642}}}
#metrics {"StartTime": 1655370523.396259, "EndTime": 1655370523.3963413, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 26}, "Metrics": {"validation_mse_objective": {"sum": 61.40277458639706, "count": 1, "min": 61.40277458639706, "max": 61.40277458639706}}}
#metrics {"StartTime": 1655370523.3963935, "EndTime": 1655370523.3964741, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 27}, "Metrics": {"validation_mse_objective": {"sum": 61.177875593596816, "count": 1, "min": 61.177875593596816, "max": 61.177875593596816}}}
#metrics {"StartTime": 1655370523.3965266, "EndTime": 1655370523.396607, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 28}, "Metrics": {"validation_mse_objective": {"sum": 61.88241277956495, "count": 1, "min": 61.88241277956495, "max": 61.88241277956495}}}
#metrics {"StartTime": 1655370523.3966587, "EndTime": 1655370523.3967392, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 29}, "Metrics": {"validation_mse_objective": {"sum": 62.56720329733456, "count": 1, "min": 62.56720329733456, "max": 62.56720329733456}}}
#metrics {"StartTime": 1655370523.396791, "EndTime": 1655370523.3968744, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 30}, "Metrics": {"validation_mse_objective": {"sum": 61.944091796875, "count": 1, "min": 61.944091796875, "max": 61.944091796875}}}
#metrics {"StartTime": 1655370523.396927, "EndTime": 1655370523.397007, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "model": 31}, "Metrics": {"validation_mse_objective": {"sum": 62.8279478783701, "count": 1, "min": 62.8279478783701, "max": 62.8279478783701}}}
[06/16/2022 09:08:43 INFO 139637139089216] #quality_metric: host=algo-1, epoch=13, validation mse_objective <loss>=13.984094357958027
[06/16/2022 09:08:43 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=13, criteria=mse_objective, value=13.450200697954964
[06/16/2022 09:08:43 INFO 139637139089216] Saving model for epoch: 13
[06/16/2022 09:08:43 INFO 139637139089216] Saved checkpoint to "/tmp/tmp5rbsw0hu/mx-mod-0000.params"
[06/16/2022 09:08:43 INFO 139637139089216] Early stop condition met. Stopping training.
[06/16/2022 09:08:43 INFO 139637139089216] #progress_metric: host=algo-1, completed 100 % epochs
#metrics {"StartTime": 1655370522.8730938, "EndTime": 1655370523.4062154, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training", "epoch": 13, "Meta": "training_data_iter"}, "Metrics": {"Total Records Seen": {"sum": 6855.0, "count": 1, "min": 6855, "max": 6855}, "Total Batches Seen": {"sum": 241.0, "count": 1, "min": 241, "max": 241}, "Max Records Seen Between Resets": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Max Batches Seen Between Resets": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Reset Count": {"sum": 16.0, "count": 1, "min": 16, "max": 16}, "Number of Records Since Last Reset": {"sum": 455.0, "count": 1, "min": 455, "max": 455}, "Number of Batches Since Last Reset": {"sum": 16.0, "count": 1, "min": 16, "max": 16}}}
[06/16/2022 09:08:43 INFO 139637139089216] #throughput_metric: host=algo-1, train throughput=853.2635130392172 records/second
[06/16/2022 09:08:43 WARNING 139637139089216] wait_for_all_workers will not sync workers since the kv store is not running distributed
[06/16/2022 09:08:43 WARNING 139637139089216] wait_for_all_workers will not sync workers since the kv store is not running distributed
[06/16/2022 09:08:43 INFO 139637139089216] #early_stopping_criteria_metric: host=algo-1, epoch=13, criteria=mse_objective, value=13.450200697954964
[06/16/2022 09:08:43 INFO 139637139089216] #validation_score (algo-1) : ('mse_objective', 12.65540687710631)
[06/16/2022 09:08:43 INFO 139637139089216] #validation_score (algo-1) : ('mse', 12.65540687710631)
[06/16/2022 09:08:43 INFO 139637139089216] #validation_score (algo-1) : ('absolute_loss', 2.5288796518363204)
[06/16/2022 09:08:43 INFO 139637139089216] #validation_score (algo-1) : ('rmse', 3.557443868440697)
[06/16/2022 09:08:43 INFO 139637139089216] #validation_score (algo-1) : ('r2', 0.7973001536506935)
[06/16/2022 09:08:43 INFO 139637139089216] #validation_score (algo-1) : ('mae', 2.5288796144373276)
[06/16/2022 09:08:43 INFO 139637139089216] #quality_metric: host=algo-1, validation mse_objective <loss>=12.65540687710631
[06/16/2022 09:08:43 INFO 139637139089216] #quality_metric: host=algo-1, validation mse <loss>=12.65540687710631
[06/16/2022 09:08:43 INFO 139637139089216] #quality_metric: host=algo-1, validation absolute_loss <loss>=2.5288796518363204
[06/16/2022 09:08:43 INFO 139637139089216] #quality_metric: host=algo-1, validation rmse <loss>=3.557443868440697
[06/16/2022 09:08:43 INFO 139637139089216] #quality_metric: host=algo-1, validation r2 <loss>=0.7973001536506935
[06/16/2022 09:08:43 INFO 139637139089216] #quality_metric: host=algo-1, validation mae <loss>=2.5288796144373276
[06/16/2022 09:08:43 INFO 139637139089216] Best model found for hyperparameters: {"optimizer": "adam", "learning_rate": 0.1, "wd": 0.0001, "l1": 0.0, "lr_scheduler_step": 10, "lr_scheduler_factor": 0.99, "lr_scheduler_minimum_lr": 1e-05}
[06/16/2022 09:08:43 INFO 139637139089216] Saved checkpoint to "/tmp/tmpx0uodk2f/mx-mod-0000.params"
[06/16/2022 09:08:43 INFO 139637139089216] Test data is not provided.
#metrics {"StartTime": 1655370513.1834052, "EndTime": 1655370523.4940147, "Dimensions": {"Algorithm": "Linear Learner", "Host": "algo-1", "Operation": "training"}, "Metrics": {"initialize.time": {"sum": 421.53167724609375, "count": 1, "min": 421.53167724609375, "max": 421.53167724609375}, "epochs": {"sum": 15.0, "count": 1, "min": 15, "max": 15}, "check_early_stopping.time": {"sum": 10.078668594360352, "count": 15, "min": 0.22172927856445312, "max": 1.9156932830810547}, "update.time": {"sum": 9697.13544845581, "count": 14, "min": 527.1692276000977, "max": 934.0167045593262}, "finalize.time": {"sum": 78.68623733520508, "count": 1, "min": 78.68623733520508, "max": 78.68623733520508}, "setuptime": {"sum": 43.54071617126465, "count": 1, "min": 43.54071617126465, "max": 43.54071617126465}, "totaltime": {"sum": 10659.562349319458, "count": 1, "min": 10659.562349319458, "max": 10659.562349319458}}}

2022-06-16 09:08:50 Uploading - Uploading generated training model
2022-06-16 09:09:24 Completed - Training job completed
Training seconds: 132
Billable seconds: 132

Great! The training is done but it has also generated a lot of logs in the output. These logs are also available in the CloudWatch and if you want you can disable them from the fit function using logs='None' parameter. Let’s try to analyze what information is presented in these logs.

The first part of these logs is related to infrastructure provisioning, downloading the training container, the data, and starting the training run.

2022-06-15 10:19:35 Starting - Starting the training job...
2022-06-15 10:19:51 Starting - Preparing the instances for trainingProfilerReport-1655288375: InProgress ......
2022-06-15 10:21:04 Downloading - Downloading input data......
2022-06-15 10:21:50 Training - Downloading the training image...
2022-06-15 10:22:35 Training - Training image download completed. Training in progress...

The second part of these logs is related to training performance metrics. The majority of the logs belong to this part.

[06/15/2022 10:22:53 INFO 140271381174080] #validation_score (algo-1) : ('mse_objective', 12.65540687710631)
[06/15/2022 10:22:53 INFO 140271381174080] #validation_score (algo-1) : ('mse', 12.65540687710631)
[06/15/2022 10:22:53 INFO 140271381174080] #validation_score (algo-1) : ('absolute_loss', 2.5288796518363204)
[06/15/2022 10:22:53 INFO 140271381174080] #validation_score (algo-1) : ('rmse', 3.557443868440697)
[06/15/2022 10:22:53 INFO 140271381174080] #validation_score (algo-1) : ('r2', 0.7973001536506935)
[06/15/2022 10:22:53 INFO 140271381174080] #validation_score (algo-1) : ('mae', 2.5288796144373276)

The third and final part is telling us about the Training job status and billable time in seconds.

2022-06-16 09:08:50 Uploading - Uploading generated training model
2022-06-16 09:09:24 Completed - Training job completed
Training seconds: 132
Billable seconds: 132

In our case, the billable seconds are 132. This billable time is for the ml.m5.large instance that we have configured for our training run. To find the billable amount we first need to find the price rate for our selected machine. For this go to the SageMaker pricing link https://aws.amazon.com/sagemaker/pricing/ and select On Demand Pricing. From the given tabs click on the Training tab. Select your region US East (N. Virginia). This will show you the pricing of different training instances. This page can also be used to find the available training instance types in your region. From this page, we can find that the price for our instance type ml.m5.large is $0.115. This price rate is per hour so we need to convert our billable time to per hour before multiplying it. i.e. (132/3600) * 0.115 = $0.0042 which is less than a penny for our training job.

We can also use Estimator object to get important information related to our training run.

##
# to get the output path where the trained model artifacts are stored
ll_estimator.output_path

's3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/output/'

##
# to get the training job name
ll_estimator.latest_training_job.job_name

'linear-learner-2022-06-16-09-04-57-576'

##
# to get the trained model location
ll_estimator.model_data

's3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/output/linear-learner-2022-06-16-09-04-57-576/output/model.tar.gz'

#collapse-output
# to get all the configuration information for our training run
ll_estimator.latest_training_job.describe()

{'TrainingJobName': 'linear-learner-2022-06-16-09-04-57-576',
 'TrainingJobArn': 'arn:aws:sagemaker:us-east-1:801598032724:training-job/linear-learner-2022-06-16-09-04-57-576',
 'ModelArtifacts': {'S3ModelArtifacts': 's3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/output/linear-learner-2022-06-16-09-04-57-576/output/model.tar.gz'},
 'TrainingJobStatus': 'Completed',
 'SecondaryStatus': 'Completed',
 'HyperParameters': {'mini_batch_size': '30', 'predictor_type': 'regressor'},
 'AlgorithmSpecification': {'TrainingImage': '382416733822.dkr.ecr.us-east-1.amazonaws.com/linear-learner:1',
  'TrainingInputMode': 'File',
  'MetricDefinitions': [{'Name': 'train:progress',
    'Regex': '#progress_metric: host=\\S+, completed (\\S+) %'},
   {'Name': 'validation:mae',
    'Regex': '#quality_metric: host=\\S+, validation mae <loss>=(\\S+)'},
   {'Name': 'train:objective_loss',
    'Regex': '#quality_metric: host=\\S+, epoch=\\S+, train \\S+_objective <loss>=(\\S+)'},
   {'Name': 'test:objective_loss',
    'Regex': '#quality_metric: host=\\S+, test \\S+_objective <loss>=(\\S+)'},
   {'Name': 'test:mse',
    'Regex': '#quality_metric: host=\\S+, test mse <loss>=(\\S+)'},
   {'Name': 'validation:objective_loss',
    'Regex': '#quality_metric: host=\\S+, epoch=\\S+, validation \\S+_objective <loss>=(\\S+)'},
   {'Name': 'validation:objective_loss:final',
    'Regex': '#quality_metric: host=\\S+, validation \\S+_objective <loss>=(\\S+)'},
   {'Name': 'validation:rmse',
    'Regex': '#quality_metric: host=\\S+, validation rmse <loss>=(\\S+)'},
   {'Name': 'test:absolute_loss',
    'Regex': '#quality_metric: host=\\S+, test absolute_loss <loss>=(\\S+)'},
   {'Name': 'validation:mse',
    'Regex': '#quality_metric: host=\\S+, validation mse <loss>=(\\S+)'},
   {'Name': 'train:mse',
    'Regex': '#quality_metric: host=\\S+, train mse <loss>=(\\S+)'},
   {'Name': 'train:absolute_loss',
    'Regex': '#quality_metric: host=\\S+, train absolute_loss <loss>=(\\S+)'},
   {'Name': 'train:objective_loss:final',
    'Regex': '#quality_metric: host=\\S+, train \\S+_objective <loss>=(\\S+)'},
   {'Name': 'validation:r2',
    'Regex': '#quality_metric: host=\\S+, validation r2 <loss>=(\\S+)'},
   {'Name': 'train:throughput',
    'Regex': '#throughput_metric: host=\\S+, train throughput=(\\S+) records/second'},
   {'Name': 'validation:absolute_loss',
    'Regex': '#quality_metric: host=\\S+, validation absolute_loss <loss>=(\\S+)'}],
  'EnableSageMakerMetricsTimeSeries': False},
 'RoleArn': 'arn:aws:iam::801598032724:role/service-role/AmazonSageMaker-ExecutionRole-20220516T161743',
 'InputDataConfig': [{'ChannelName': 'train',
   'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',
     'S3Uri': 's3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/input/training/training_data.csv',
     'S3DataDistributionType': 'FullyReplicated'}},
   'ContentType': 'text/csv',
   'CompressionType': 'None',
   'RecordWrapperType': 'None'},
  {'ChannelName': 'validation',
   'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',
     'S3Uri': 's3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/input/validation_data/validation_data.csv',
     'S3DataDistributionType': 'FullyReplicated'}},
   'ContentType': 'text/csv',
   'CompressionType': 'None',
   'RecordWrapperType': 'None'}],
 'OutputDataConfig': {'KmsKeyId': '',
  'S3OutputPath': 's3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/output/'},
 'ResourceConfig': {'InstanceType': 'ml.m5.large',
  'InstanceCount': 1,
  'VolumeSizeInGB': 30},
 'StoppingCondition': {'MaxRuntimeInSeconds': 86400},
 'CreationTime': datetime.datetime(2022, 6, 16, 9, 4, 57, 790000, tzinfo=tzlocal()),
 'TrainingStartTime': datetime.datetime(2022, 6, 16, 9, 6, 48, 358000, tzinfo=tzlocal()),
 'TrainingEndTime': datetime.datetime(2022, 6, 16, 9, 9, 0, 598000, tzinfo=tzlocal()),
 'LastModifiedTime': datetime.datetime(2022, 6, 16, 9, 9, 45, 67000, tzinfo=tzlocal()),
 'SecondaryStatusTransitions': [{'Status': 'Starting',
   'StartTime': datetime.datetime(2022, 6, 16, 9, 4, 57, 790000, tzinfo=tzlocal()),
   'EndTime': datetime.datetime(2022, 6, 16, 9, 6, 48, 358000, tzinfo=tzlocal()),
   'StatusMessage': 'Preparing the instances for training'},
  {'Status': 'Downloading',
   'StartTime': datetime.datetime(2022, 6, 16, 9, 6, 48, 358000, tzinfo=tzlocal()),
   'EndTime': datetime.datetime(2022, 6, 16, 9, 7, 44, 44000, tzinfo=tzlocal()),
   'StatusMessage': 'Downloading input data'},
  {'Status': 'Training',
   'StartTime': datetime.datetime(2022, 6, 16, 9, 7, 44, 44000, tzinfo=tzlocal()),
   'EndTime': datetime.datetime(2022, 6, 16, 9, 8, 50, 28000, tzinfo=tzlocal()),
   'StatusMessage': 'Training image download completed. Training in progress.'},
  {'Status': 'Uploading',
   'StartTime': datetime.datetime(2022, 6, 16, 9, 8, 50, 28000, tzinfo=tzlocal()),
   'EndTime': datetime.datetime(2022, 6, 16, 9, 9, 0, 598000, tzinfo=tzlocal()),
   'StatusMessage': 'Uploading generated training model'},
  {'Status': 'Completed',
   'StartTime': datetime.datetime(2022, 6, 16, 9, 9, 0, 598000, tzinfo=tzlocal()),
   'EndTime': datetime.datetime(2022, 6, 16, 9, 9, 0, 598000, tzinfo=tzlocal()),
   'StatusMessage': 'Training job completed'}],
 'FinalMetricDataList': [{'MetricName': 'train:progress',
   'Value': 100.0,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'validation:mae',
   'Value': 2.5288796424865723,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'train:objective_loss',
   'Value': 0.28216010332107544,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'validation:objective_loss',
   'Value': 13.984094619750977,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'validation:objective_loss:final',
   'Value': 12.655406951904297,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'validation:rmse',
   'Value': 3.557443857192993,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'validation:mse',
   'Value': 12.655406951904297,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'validation:r2',
   'Value': 0.7973001599311829,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'train:throughput',
   'Value': 853.2634887695312,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())},
  {'MetricName': 'validation:absolute_loss',
   'Value': 2.5288796424865723,
   'Timestamp': datetime.datetime(2022, 6, 16, 9, 8, 43, tzinfo=tzlocal())}],
 'EnableNetworkIsolation': False,
 'EnableInterContainerTrafficEncryption': False,
 'EnableManagedSpotTraining': False,
 'TrainingTimeInSeconds': 132,
 'BillableTimeInSeconds': 132,
 'ProfilerConfig': {'S3OutputPath': 's3://sagemaker-us-east-1-801598032724/2022-06-08-sagemaker-training-overview/output/',
  'ProfilingIntervalInMilliseconds': 500},
 'ProfilerRuleConfigurations': [{'RuleConfigurationName': 'ProfilerReport-1655370297',
   'RuleEvaluatorImage': '503895931360.dkr.ecr.us-east-1.amazonaws.com/sagemaker-debugger-rules:latest',
   'VolumeSizeInGB': 0,
   'RuleParameters': {'rule_to_invoke': 'ProfilerReport'}}],
 'ProfilerRuleEvaluationStatuses': [{'RuleConfigurationName': 'ProfilerReport-1655370297',
   'RuleEvaluationJobArn': 'arn:aws:sagemaker:us-east-1:801598032724:processing-job/linear-learner-2022-06-16--profilerreport-1655370297-d239ec9a',
   'RuleEvaluationStatus': 'NoIssuesFound',
   'LastModifiedTime': datetime.datetime(2022, 6, 16, 9, 9, 45, 61000, tzinfo=tzlocal())}],
 'ProfilingStatus': 'Enabled',
 'ResponseMetadata': {'RequestId': 'ae027c8b-2090-40eb-b904-75509ee9f3e5',
  'HTTPStatusCode': 200,
  'HTTPHeaders': {'x-amzn-requestid': 'ae027c8b-2090-40eb-b904-75509ee9f3e5',
   'content-type': 'application/x-amz-json-1.1',
   'content-length': '6459',
   'date': 'Thu, 16 Jun 2022 09:10:08 GMT'},
  'RetryAttempts': 0}}

Training logs can also be viewed on AWS CloudWatch under LogGroup /aws/sagemaker/TrainingJobs and Log Stream same as the training job name.

Deploy Model and Make Predictions

Our model is trained and now we can deploy it to AWS SageMaker endpoint with a single call. These endpoints are fully managed and support autoscaling.

In the deployment call we are required to provide * instance type this is the machine type on which we want to deploy our model * instance count number of instances we want to provision for our machine. If more than one instance is provisioned then SageMaker EndPoint will automatically load balance between them * endpoint_name a unique identifier for your endpoint. If it is not provided then it will default to your training job name.

It is important to note that endpoints are scoped to an individual AWS account, and are not public. The URL does not contain the account ID, but SageMaker determines the account ID from the authentication token that is supplied by the caller. We will discuss in a separate post on how to use AWS Lambda and API Gateway to set up and deploy your inference endpoint for public access.

Read more about deployment from the documentation sagemaker.estimator.Estimator.deploy

##
# define the endpoint. can be any unique string
endpoint_name = ll_estimator.latest_training_job.job_name

print(endpoint_name)

linear-learner-2022-06-16-09-04-57-576

##
# deploy the model
ll_predictor = ll_estimator.deploy(
    initial_instance_count=1, 
    instance_type='ml.t2.medium',
    endpoint_name=endpoint_name
)

-------------!

This call will take a minute or so to spin up the required machine and create the endpoint. Once it is ready it will be available in SageMaker Resources / Endpoints.

##
# a test sample to get live inference from our model
test_sample = '0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,4.98'

Similar to data input channels that tell the Estimator object on how to read the data from S3 bucket, we need to tell our predictor object on how to receive (encode) input and return (decode) data during inference. For this we will use sagemaker.serializers.CSVSerializer object to serialize data of various formats to a CSV-formatted string.

from sagemaker.serializers import CSVSerializer
from sagemaker.deserializers import CSVDeserializer

ll_predictor.serializer = CSVSerializer()
ll_predictor.deserializer = CSVDeserializer()

Note that we can also pass these Serializers in the deploy method call like this

ll_predictor = ll_estimator.deploy(
    initial_instance_count=1, 
    instance_type='ml.t2.medium',
    endpoint_name=endpoint_name,
    serializer = CSVSerializer(),
    deserializer = CSVDeserializer()
)

##
# call the endpoint for live inference
ll_predictor.predict(test_sample)

[['29.98671531677246']]

The prediction output tells us that this house will cost $29986.

When we called the estimator.deploy() it has returned us sagemaker.Predictor object. If we want to call endpoint from a separate code then we can use the same ‘Predictor’ object to call it. You can read more about this class here sagemaker.predictor.Predictor

from sagemaker.predictor import Predictor

ll_predictor_2 = Predictor(
    endpoint_name, 
    session, # If not specified, then one is created using the default AWS configuration chain
    CSVSerializer(), 
    CSVDeserializer()
)

ll_predictor_2.predict(test_sample)

[['29.98671531677246']]

Clean Up

Remember that the inference machine will remain in use until you take it down yourself. This is unlike training where provisioning and decommissioning of machines are done automatically. We have used ml.t2.medium for hosting our model which costs around $0.056 per hour. This is not much but still, do clean up your environment once you are done experimenting to avoid unnecessary bills.

You can remove the endpoint with a single call.

##
# ll_predictor_2 will work too

ll_predictor.delete_endpoint()

Summary

In this post we have learned about the SageMaker workflow, and how to train and deploy machine learning models with a handful of APIs from SageMaker SDK.