Applications - Feedback

Feedback is used to train and improve the performance of the Application. Feedback is a human task, where users can review predictions from an Application and correct any wrong predictions. Feedback is automatically scheduled by the Cogniac system, surfacing ambiguous model predictions to users for additional feedback, however feedback can be provided directly with a properly formatted feedback response message.

The Cogniac mobile app and web app have an optimized feedback interface, and we advise users to utilize the existing interfaces for high-volume feedback tasks (particularly early in the Application training process).

Feedback Request Object

A system-scheduled feedback request message will contain a media object, an optional focus area, an optional list of other media objects related to the model assertion, and a list of subject associations.

Fields	Example	Description
media_id string	"media_id_123"	Unique ID automatically assigned to each media object upon creation.
media map		A Cogniac media object
subjects array		List of subject media associations
other_media array		(optional) List of other media items related to the model assertion.
focus map	{ "box": {"x0": 50, "x1": 275, "y0": 634, "y1": 870}}	(optional) single box_region input focus area

Feedback Response Object

A user-provided feedback response message must contain a media object, a list of subject-media assertions, an optional focus area, an optional list of other media objects related to the model assertion, and optional device and response metadata.

Fields	Example	Description
media_id string	"media_id_123"	Unique ID automatically assigned to each media object upon creation.
media map		A Cogniac media object
subjects array		List of user subject media assertions.
other_media array		(optional) List of other media items related to the model assertion.
focus map	{ "box": {"x0": 50, "x1": 275, "y0": 634, "y1": 870}}	(optional) Single box_region input focus area
client_data map	{"device": "iPhone 7", "sw_version": 10}	(optional) Map of client device and application metadata.
decision_time float	0.123	(optional) Client feedback time.

User Subject-Media Assertion Object

When providing feedback, a user must submit a list of subject-media assertions of the following form:

Fields	Example	Description
subject_uid string	"cat_123"	The unique ID of the subject the user is associating with the media item.
result string	"True"	True - the subject is associated with the media False - the subject is NOT associated with the media Note that not all application types support each of these results. Please see the app-specific semantics defined below.
app_data_type string	"box_set"	(optional) Optional type of extra app-specific data for certain app types.
app_data object	{ "box": {"x0": 50, "x1": 275, "y0": 634, "y1": 870}, "probability": 0.85}	Optional extra app-specific data (see Cogniac Applications for more information on app_data formatting.)

Feedback Semantics

The semantics of feedback request and response messages vary depending on the type of application. Feedback messages utilize the concept of application-specific data to refine proposed subject-media associations. For more information see Application-Specific Data

📘
Feedback by Application Type
Classification
Feedback Request
System proposes single most likely subject_uid associated with a full-frame media_id.
Feedback Response
User supplies single most likely subject_uid associated with a media_id with result="True".
False feedback is not allowed for classification applications.
Detection Fullframe
Feedback Request
System proposes probability of association for each output subject_uid configured for an application.
Feedback Response
User supplies result="True" or result="False" for each output subject_uid configured for an application in a single feedback message.
Detection
Feedback Request
System proposes a list of most likely box set associated with the application’s subject_uid within a media_id.
Feedback Response
Users supply a single feedback response for a given media media_id (optionally within a focus area) with result="True", app_data_type="box_set", and app_data containing a list of box_set regions.
In the case that there are no true instances of the subject_uid present in an image then result="False" feedback should be provided on the subject_uid without app_data or app_data_type specified. This case is essentially identical to the detection fullframe app False case where we are asserting that the subject is not present in the image.
Static Count
Feedback Request
System proposes single most likely count associated with the application’s subject_uid within a media_id.
Feedback Response
Users supply a single feedback response for a given media media_id (optionally within a focus area) which contains either result="True", app_data_type="count", with the correct count in app_data, or result="False", with no app_data or app_data_type.
Offset_x, Offset_y,
Feedback Request
System proposes single most likely offset associated with the application’s subject_uid within a media_id. The offset is measured from the reference_edge x or y as specified in the application configuration . A line may be drawn on the image to illustrate the offset from the edge
Feedback Response
Users supply a single feedback response for a given media media_id (optionally within a focus area) which contains either result="True", app_data_type="offset_x" (or offset_y), with the correct offset value in app_data, or result="False", with no app_data or app_data_type.
Point
Feedback Request
System proposes single most likely point associated with the application’s subject_uid within a media_id.
Feedback Response
Users supply a single feedback response for a given media media_id (optionally within a focus area) which contains either result="True", app_data_type="point", with the correct point definition in app_data, or result="False", with no app_data or app_data_type.

Retrieving Feedback Requests

The Application Feedback endpoint can be used to retrieve feedback requests that have been previously scheduled by the Cogniac System.

🚧
Retrieving Feedback
Once retrieved, feedback requests are removed from the application's feedback queue and cannot be accessed by other users.

📘
Refreshing Scheduled Feedback
Feedback that is scheduled by the Cogniac System can become out-of-date if a new model is released for the given application. To automatically reprocess scheduled feedback through any newly released models, set refresh_feedback to True in the application's configuration.

Argument	Example	Description
limit boolean	5	(optional) (optional limit between 1 and 10 max, defaults to 10)

GET /1/applications/{application_id}/feedback
Host: https://api.cogniac.io

Example: Retrieve Scheduled Feedback

curl -X GET https://api.cogniac.io/1/applications/di71rG94/feedback?limit=5 \
-H "Authorization: Bearer abcdefg.hijklmnop.qrstuvwxyz"

import cogniac

# connect to the tenant
cc = cogniac.CogniacConnection(username="[email protected]", 
                               password="myPassword", 
                               tenant_id="63QhzFLc9tg4")

# get the app object
my_app = cc.get_application('di71rG94')

# the detections method returns an iterator of app detections
feedback = my_app.get_feedback(limit=5)
for fb in feedback:
  print fb

The response is a list of feedback requests:

[
  {    
    "media": {
      "author": null,
      "author_profile_url": null,
      "created_at": 1497662135.660577,
      "external_media_id": null,
      "filename": null,
      "frame": null,
      "image_height": 1200,
      "image_width": 800,
      "license": null,
      "md5": "05180dac616104ff004027a5f274d6ad",
      "media_format": "JPEG",
      "media_id": "XVLNY8FEKGT1VZC3XN2JGS2ZVAWH",
      "media_src": "twitter",
      "media_timestamp": 1497662135.629489,
      "media_url": "https://cogniac-media.s3.amazonaws.com/XVLNY8FEKGT1VZC3XN2JGS2ZVAWH",
      "meta_tags": ["twitter"],
      "network_camera_id": null,
      "original_landing_url": null,
      "original_url": null,
      "parent_media_id": null,
      "parent_media_ids": [],
      "preview_url": null,
      "resize_urls": {
        "454": "https://cogniac-media.s3.amazonaws.com/XVLNY8FEKGT1VZC3XN2JGS2ZVAWH_max_454",
        "750": "https://cogniac-media.s3.amazonaws.com/XVLNY8FEKGT1VZC3XN2JGS2ZVAWH_max_750"
       },
      "set_assignment": "training",
      "size": 168423,
      "source_url": null,
      "status": "success",
      "tenant_id": "skjdfh853k4jt",
      "title": null,
      "uploaded_by_user": null,
      "video": false
    },
    "focus":{
      "box": {"x0": 50, "x1": 275, "y0": 634, "y1": 870}
    },
        
    "subjects": [
      {
        "app_data_type": null,
        "subject_uid": "bill",
        "media_id": "eb51e7275210d74f5a0116e32c8b32d4",
        "probability": 0.9424338417689073,
        "app_data": null
      },
      {
        "app_data_type": null,
        "subject_uid": "prashant",
        "media_id": "eb51e7275210d74f5a0116e32c8b32d4",
        "probability": 0.01827074101387713,
        "app_data": null
      },
      {
        "app_data_type": null,
        "subject_uid": "amy",
        "media_id": "eb51e7275210d74f5a0116e32c8b32d4",
        "probability": 0.023146493553089783,
        "app_data": null
      },
      {
        "app_data_type": null,
        "subject_uid": "visitor",
        "media_id": "eb51e7275210d74f5a0116e32c8b32d4",
        "probability": 0.02874867741689087,
        "app_data": null
      }
    ],
    "media_id": "eb51e7275210d74f5a0116e32c8b32d4"
  }
]

Providing Feedback

An application's Feedback endpoint can be used to provide feedback on subject-media associations in the context of that application.

📘
Providing Feedback
A media item does not need to be scheduled for user feedback by the Cogniac system in order to provide a user assertion on a subject-media association.

POST /1/applications/{application_id}/feedback
Host: https://api.cogniac.io
Returns: 204 (empty response)

Example: Provide Application Feedback

curl -X POST https://api.cogniac.io/1/applications/di71rG94/feedback \
-H "Content-Type: application/json" \
-H "Authorization: Bearer abcdefg.hijklmnop.qrstuvwxyz" \
-d '{
"media_id":"29djlsd",
"client_data":{},
"decision_time":2940249,
"subjects":
  [
    {
  "subject_uid":"cat",
  "result":"True"
    } 
  ]
}'

import cogniac

# connect to the tenant
cc = cogniac.CogniacConnection(username="[email protected]", 
                               password="myPassword", 
                               tenant_id="63QhzFLc9tg4")

# get the app object
my_app = cc.get_application('di71rG94')

# the detections method returns an iterator of app detections
my_app.post_feedback(media_id="29djlsd",
                  subjects=[{"subject_uid":"cat",
                             "result":"True"
                            }])
# returns None

Pending Feedback Count

An application's Pending Feedback endpoint can be used to return the total length of the scheduled feedback queue.

GET /1/applications/{application_id}/feedback/pending
Host: https://api.cogniac.io

Example: Retrieve Scheduled Feedback Request Count

curl -X POST https://api.cogniac.io/1/applications/di71rG94/feedback/pending \
-H "Authorization: Bearer abcdefg.hijklmnop.qrstuvwxyz"

import cogniac

# connect to the tenant
cc = cogniac.CogniacConnection(username="[email protected]", 
                               password="myPassword", 
                               tenant_id="63QhzFLc9tg4")

# get the app object
my_app = cc.get_application('di71rG94')

# the detections method returns an iterator of app detections
n = app.pending_feedback()
print n

Applications - Feedback

Feedback Request Object

Feedback Response Object

User Subject-Media Assertion Object

Feedback Semantics

📘
Feedback by Application Type

Retrieving Feedback Requests

🚧
Retrieving Feedback

📘
Refreshing Scheduled Feedback

Example: Retrieve Scheduled Feedback

Providing Feedback

📘
Providing Feedback

Example: Provide Application Feedback

Pending Feedback Count

Example: Retrieve Scheduled Feedback Request Count

Feedback Request Object

Feedback Response Object

User Subject-Media Assertion Object

Feedback Semantics

📘Feedback by Application Type

Retrieving Feedback Requests

🚧Retrieving Feedback

📘Refreshing Scheduled Feedback

Example: Retrieve Scheduled Feedback

Providing Feedback

📘Providing Feedback

Example: Provide Application Feedback

Pending Feedback Count

Example: Retrieve Scheduled Feedback Request Count

📘
Feedback by Application Type

🚧
Retrieving Feedback

📘
Refreshing Scheduled Feedback

📘
Providing Feedback