Applications - Performance
Three application endpoints return performance statistics such as precision, recall, and F1 score for all previously released models calculated against different data sets.
All performance endpoints accept the same query arguments and return data in the same format.
Performance Request Body
| Argument | Example | Definition |
|---|---|---|
| start unix timestamp | 1494885870 | (optional) Earliest model release timestamp which to return performance results. Defaults to 0. |
| end unix timestamp | 1494886421 | (optional) Latest model release timestamp which to return performance results. Defaults to current time. |
| limit integer | 10 | (optional) Maximum number of records to return. Defaults to returning all available records. |
| reverse boolean | True | (optional) True - return records in descending time order. False - return records in ascending time order. Defaults to False. |
Application Performance Response
| Fields | Example | Definition |
|---|---|---|
| app_id string | "di71rG94" | Unique ID of the application. |
| data_count integer | 347 | Total number of validation data tested against the model. |
| loss float | 0.09 | The error rate of the model. |
| model_performance map | A model performance map. | |
| model_performance_per_subject array | An array of model performance maps for each output subject. | |
| output_subjects list | ["cat", "dog"] | The list of application output subject_uids. |
| release_metrics string | "best_F1" | The statistic used to release improved models. |
| release_time unix timestamp | 1234567890 | Original model release timestamp. |
| type string | "classification" | Application type. |
| updated_at unix timestamp | 1234567890 | The time the model metrics were updated. |
Model Performance Object
| Field | Definition |
|---|---|
| FN integer | False Negatives, the total count of subject-media associations in the validation set incorrectly missed. I.e. the model did not detect a subject present in the media. |
| FP integer | False Positives, the total count of subject-media associations in the validation set incorrectly detected. I.e. the model detected a subject not present in the media. |
| TN integer | True Negatives, the total count of subject-media associations correctly negatively identified. I.e. the model correctly negatively associated the subject with the media. |
| TP integer | True Positives, the total count of subject-media associations correctly identified in the validation set. I.e. the model correctly identified the subject in the media. |
| accuracy float | A measure of overall accuracy of the model. Measured as: |
| precision float | The fraction of positive subject-media assertions made by the model that are correct. Measured as: |
| recall float | The fraction of positive subject-media associations detected by the model. Measured as: |
| F1 float | An overall measure of the model's accuracy. The harmonic average of precision and recall. Measured as: |
Release Validation Performance
Release Validation performance reports model performance scores tested against the validation data set at the time each model was released.
GET /1/applications/{application_id}/performance/releaseValidation
Host: https://api.cogniac.io
Current Validation Performance
Current Validation performance reports model performance scores tested against the current validation data set.
GET /1/applications/{application_id}/performance/currentValidation
Host: https://api.cogniac.io
New Random Performance
New Random performance reports model performance against a set of data that each model has not encountered in either a test or validation set at the time of mode release.
GET /1/applications/{application_id}/performance/newRandom
Host: https://api.cogniac.io
Example: Retrieve Application Performance Against Current Validation Set
curl -X GET https://api.cogniac.io/1/applications/di71rG94/performance/currentValidation?limit=10 \
-H "Authorization: Bearer abcdefg.hijklmnop.qrstuvwxyz"
import requests
my_headers = {"Authorization":"Bearer abcdefg.hijklmnop.qrstuvwxyz"}
resp = requests.get("https://api.cogniac.io/1/applications/di71rG94/performance/currentValidation?limit=10",
headers=my_headers)
print resp
{
"data": [
{
"app_id": "di71rG94",
"data_count": 54,
"loss": 0.2407407407407407,
"model_performance": {
"FP": 13,
"F1": 0.7592592592592593,
"recall": 0.7592592592592593,
"precision": 0.7592592592592593,
"TP": 41,
"TN": 41,
"FN": 13,
"accuracy": 0.7592592592592593
},
"model_performance_per_subject": {
"cat": {
"FP": 7,
"F1": 0.8354430379746836,
"recall": 0.8461538461538461,
"precision": 0.825,
"TP": 33,
"TN": 8,
"FN": 6,
"accuracy": 0.7592592592592593
},
"dog": {
"FP": 6,
"F1": 0.5517241379310344,
"recall": 0.5333333333333333,
"precision": 0.5714285714285714,
"TP": 8,
"TN": 33,
"FN": 7,
"accuracy": 0.7592592592592593
}
},
"output_subjects": [
"cat",
"dog"
],
"release_metrics": "best_F1",
"release_time": 1484251629.765,
"type": "classification",
"updated_at": 1484251629.765
},
{
"app_id": "di71rG94",
"data_count": 104,
"loss": 0.25,
"model_performance": {
"FP": 26,
"F1": 0.75,
"recall": 0.75,
"precision": 0.75,
"TP": 78,
"TN": 78,
"FN": 26,
"accuracy": 0.75
},
"model_performance_per_subject": {
"cat": {
"FP": 10,
"F1": 0.8194444444444444,
"recall": 0.7866666666666666,
"precision": 0.855072463768116,
"TP": 59,
"TN": 19,
"FN": 16,
"accuracy": 0.75
},
"dog": {
"FP": 16,
"F1": 0.5937499999999999,
"recall": 0.6551724137931034,
"precision": 0.5428571428571428,
"TP": 19,
"TN": 59,
"FN": 10,
"accuracy": 0.75
}
},
"output_subjects": [
"cat",
"dog"
],
"release_metrics": "best_F1",
"release_time": 1484278580.835871,
"type": "classification",
"updated_at": 1484278580.835871
}, ...
]
}
Updated over 6 years ago