Skip to content

Available Tasks

Each task can have Inference templates and Support templates:

  • Inference templates are those templates which can be used at inference time and are those upon which the evaluation is carried out
  • Support templates are those templates which can NOT be used at inference time, since they have intrinsic assumption which requires the knowledge of the ground truth

Example of inference template:

Input Placeholder text Target Placeholder text
Recommend an item for {user_id} {target_item}

Example of support template:

Input Placeholder text Target Placeholder text
Recommend an item for {user_id}, knowing that a good item to recommend is present among {candidates} {target_item}

As you can easily see, the support template requires information that we do not have at inference time: that is, the target item to recommend

To specify one of the following tasks the .yaml file, you can simply use its name (The parsing is not case-sensitive)!

SequentialSideInfoTask

The SequentialSideInfoTask is built for AmazonDataset: the goal is to predict the next item of the order history of the user. This task has the SideInfo suffix because categories of the items bought by the user are used additional information for the prediction.

There are two different support tasks:

  • Extractive QA: For the specific user_id, given its order history and the categories of each item bought, select the next element to recommend from a list of candidates
  • Pair Seq: For the specific user_id, given only one element from the order history and its categories, predict the immediate successor of the element

Inference templates:

Template ID Input Placeholder text Target Placeholder text
0 sequential recommendation - {user_id}:

Predict for the user the next element of the following sequence -> {order_history}
The category of each element of the sequence is -> {category_history} 		{target_item}
1 sequential recommendation - {user_id}:

Predict the next element which the user will buy given the following order history -> {order_history}
Each item bought belongs to these categories (in order) -> {category_history} 		{target_item}
2 sequential recommendation - {user_id}:

What is the element that should be recommended to the user knowing that it has bought -> {order_history}
Categories of the items are -> {category_history} 		{target_item}
3 sequential recommendation - {user_id}:

Recommend to the user an item from the catalog given its order history -> {order_history}
Each item of the order history belongs to the following categories (in order) -> {category_history} 		{target_item}
4 sequential recommendation - {user_id}:

This is the order history of the user -> {order_history}
These are the categories of each item -> {category_history}
Please recommend the next element that the user will buy 		{target_item}
5 sequential recommendation - {user_id}:

Please predict what item is best to recommend to the user given its order history -> {order_history}
Categories of each item -> {category_history}
{target_item}

Extractive QA templates

Template ID Input Placeholder text Target Placeholder text
6 sequential recommendation - {user_id}:

The user has the following order history -> {order_history}
The categories of each item bought are -> {category_history}
Which item would the user buy next? Select from the following:
{candidate_items} 		{target_item}
7 sequential recommendation - {user_id}:

The user has bought {order_history}, and the categories of those items are {category_history}.
Choose an item to recommend to the user selecting from:
{candidate_items} 		{target_item}

Pair Seq templates:

Template ID Input Placeholder text Target Placeholder text
8 sequential recommendation - {user_id}:

The user has recently bought {precedent_item_id} which has the following categories: {categories_precedent_item}
What is the next item to recommend?
{target_item}
9 sequential recommendation - {user_id}:

The latest item bought by the user is {precedent_item_id}.
The categories of that item are {categories_precedent_item}.
Predict which item the user will buy next 		{target_item}

DirectSideInfoTask

The DirectSideInfoTask is built for AmazonDataset: the goal is to predict a good item to recommend for the user. This task has the SideInfo suffix because categories of the items bought by the user are used additional information for the prediction.

There are two different support tasks:

  • Extractive QA: For the specific user_id, given categories liked by the user, select the item to recommend from a list of candidates

Inference templates:

Template ID Input Placeholder text Target Placeholder text
0 direct recommendation - {user_id}:

Pick an item from the catalog knowing that these are the categories the user likes -> {unique_categories_liked}
{target_item}
1 direct recommendation - {user_id}:

Recommend an item to the user. The categories of the items bought by the user are -> {unique_categories_liked}
{target_item}
2 direct recommendation - {user_id}:

What is the item that should be recommended to the user? It likes these categories -> {unique_categories_liked}
{target_item}
3 direct recommendation - {user_id}:

Select an item to present to the user given the categories that it likes -> {unique_categories_liked}
{target_item}
4 direct recommendation - {user_id}:

These are the categories of the items bought by the user -> {unique_categories_liked}
Please recommend an item that the user will buy 		{target_item}
5 direct recommendation - {user_id}:

Please predict what item is best to recommend to the user. The categories that it likes are -> {unique_categories_liked}
{target_item}

Extractive QA templates:

Template ID Input Placeholder text Target Placeholder text
6 direct recommendation - {user_id}:

The categories liked by the user are -> {unique_categories_liked}
Which item can interest the user? Select one from the following:
{candidate_items} 		{target_item}
7 direct recommendation - {user_id}:

The user so far has bought items with these categories -> {unique_categories_liked}.
Choose an item to recommend to the user selecting from:
{candidate_items} 		{target_item}
8 direct recommendation - {user_id}:

These are the categories of the items bought by the user -> {unique_categories_liked}.
Predict an item to suggest to the user from the followings:
{candidate_items} 		{target_item}

RatingPredictionTask

The RatingPredictionTask is built for AmazonDataset: the goal is to predict the rating that the user would give to an unseen item.

Inference templates

Template ID Input Placeholder text Target Placeholder text
0 rating prediction - {user_id}:

Average rating of the user -> {avg_rating}
Continue this rating sequence for the user, predicting the rating for {item_id}:
{order_history_w_ratings} 		{target_rating}
1 rating prediction - {user_id}:

Average rating of the user -> {avg_rating}
Predict the rating that the user would give to {item_id}, by considering the following previously bought item and the rating assigned:
{order_history_w_ratings} 		{target_rating}
2 rating prediction - {user_id}:

Predict the score the user would give to {item_id} (in a 1-5 scale).
This is the user order history with associated rating that the user previously gave:
{order_history_w_ratings}
Consider that the average rating of the user is {avg_rating} 		{target_rating}
3 rating prediction - {user_id}:

This is the order history of the user with the associated rating ->
{order_history_w_ratings}
This is the average rating given by the user -> {avg_rating}
Based on that, predict the score (in a 1-5 scale) the user would give to {item_id} 		{target_rating}
4 rating prediction - {user_id}:

Please predict the user, which has an average rating of {avg_rating}, would give to {item_id} based on its order history ->
{order_history_w_ratings}
This is the average rating given by the user -> {avg_rating}
The score should be in a 1-5 scale 		{target_rating}

P5 Tasks

Below there are defined the Rating, Sequential, Direct task prompts from the P5 paper. Each task has its "Eval" counterpart, where there are defined the prompts used by the P5 author's for the evaluation phase. Each Eval task has one seen prompt (a prompt used during fine-tuning) and an unseen one.

P5RatingTask

The RatingPredictionTask is built for AmazonDataset: the goal is to predict the rating that the user would give to an unseen item.

Inference templates:

Template ID Input Placeholder text Target Placeholder text
1-1 Which star rating will user_{user_id} give item_{item_id} ? ( 1 being lowest and 5 being highest ) {star_rating}
1-2 How will user_{user_id} rate this product : {item_title} ? ( 1 being lowest and 5 being highest ) {star_rating}
1-5 Predict the user_{user_id} 's preference on item_{item_id} ( {item_title} )
-1
-2
-3
-4
-5 		{star_rating}
1-6 What star rating do you think {user_name} will give item_{item_id} ? ( 1 being lowest and 5 being highest ) {star_rating}
1-7 How will {user_name} rate this product : {item_title} ? ( 1 being lowest and 5 being highest ) {star_rating}

Support templates:

Template ID Input Placeholder text Target Placeholder text
1-3 Will user_{user_id} give item_{item_id} a {star_rating}-star rating ? ( 1 being lowest and 5 being highest ) {yes_no}
1-4 Does user_{user_id} like or dislike item_{item_id} ? {like_dislike}
1-8 Will {user_name} give a {star_rating}-star rating for {item_title} ? ( 1 being lowest and 5 being highest ) {yes_no}
1-9 Does {user_name} like or dislike {item_title} ? {like_dislike}

P5EvalRatingTask

Template ID Input Placeholder text Target Placeholder text
1-6 What star rating do you think {user_name} will give item_{item_id} ? ( 1 being lowest and 5 being highest ) {star_rating}
1-10 Predict {user_name} 's preference towards {item_title} ( 1 being lowest and 5 being highest ) {star_rating}

P5SequentialTask

The SequentialSideInfoTask is built for AmazonDataset: the goal is to predict the next item of the order history of the user. This task has the SideInfo suffix because categories of the items bought by the user are used additional information for the prediction.

Inference templates:

Template ID Input Placeholder text Target Placeholder text
2-1 Given the following purchase history of user_{user_id} :
{order_history}
predict next possible item to be purchased by the user ? 		{target_item}
2-2 I find the purchase history list of user_{user_id} :
{order_history}
I wonder what is the next item to recommend to the user . Can you help me decide ? 		{target_item}
2-3 Here is the purchase history list of user_{user_id} :
{order_history}
try to recommend next item to the user 		{target_item}
2-4 Given the following purchase history of {user_name} :
{order_history}
predict next possible item for the user 		{target_item}
2-5 Based on the purchase history of {user_name} :
{order_history}
Can you decide the next item likely to be purchased by the user ? 		{target_item}
2-6 Here is the purchase history of {user_name} :
{order_history}
What to recommend next for the user ? 		{target_item}

Extractive QAs templates:

Template ID Input Placeholder text Target Placeholder text
2-7 Here is the purchase history of user_{user_id} :
{order_history}
Select the next possible item likely to be purchased by the user from the following candidates :
{candidate_items} 		{target_item}
2-8 Given the following purchase history of {user_name} :
{order_history}
What to recommend next for the user? Select one from the following items :
{candidate_items} 		{target_item}
2-9 Based on the purchase history of user_{user_id} :
{order_history}
Choose the next possible purchased item from the following candidates :
{candidate_items} 		{target_item}
2-10 I find the purchase history list of {user_name} :
{order_history}
I wonder which is the next item to recommend to the user . Try to select one from the following candidates :
{candidate_items} 		{target_item}

Pairwise prediction templates:

Template ID Input Placeholder text Target Placeholder text
2-11 user_{user_id} has the following purchase history :
{order_history}
does the user likely to buy {target_item} next ? 		{yes_no}
2-12 According to {user_name} 's purchase history list :
{order_history}
Predict whether the user will purchase {target_item} next ? 		{yes_no}

P5EvalSequentialTask

Template ID Input Placeholder text Target Placeholder text
2-3 Here is the purchase history list of user_{user_id} :
{order_history}
try to recommend next item to the user 		{target_item}
2-13 According to the purchase history of {user_name} :
{order_history}
Can you recommend the next possible item to the user ? 		{target_item}

P5DirectTask

The P5DirectTask is built for AmazonDataset: the goal is to predict a good item to recommend for the user.

Inference templates:

Template ID Input Placeholder text Target Placeholder text
5-5 Which item of the following to recommend for {user_name} ?
{candidate_items} 		{target_item}
5-6 Choose the best item from the candidates to recommend for {user_name} ?
{candidate_items} 		{target_item}
5-7 Pick the most suitable item from the following list and recommend to user_{user_id} :
{candidate_items} 		{target_item}

Support templates:

Template ID Input Placeholder text Target Placeholder text
5-1 Will user_{user_id} likely to interact with item_{item_id} ? {yes_no}
5-2 Shall we recommend item_{item_id} to {user_name} ? {yes_no}
5-3 For {user_name}, do you think it is good to recommend {item_title} ? {yes_no}
5-4 I would like to recommend some items for user_{user_id} . Is the following item a good choice ?
{item_title} 		{yes_no}

P5EvalDirectTask

Template ID Input Placeholder text Target Placeholder text
5-5 Which item of the following to recommend for {user_name} ?
{candidate_items} 		{target_item}
5-8 We want to make recommendation for user_{user_id} . Select the best item from these candidates :
{candidate_items} 		{target_item}