Hello, I am Deepak Bhatt, and I work as a Backend Engineer in the New Lister Conversion Team. This post is for Day 24 of Mercari Advent Calendar 2022.

My team is a cross functional product team consisting of engineers, designers, product managers and data analysts. We work together to help users list more items on Mercari.

The team promotes to empower engineers and other members to take ownership of new ideas and conduct experiments to test these new ideas. Empowering members is an important concept for cross functional teams to make better products. In this article, we will talk about how we work as a cross functional team where we have created specific team processes to strive towards this goal.

How do we work in our team ?

Our team consists of around 15 members from different domains. We use scrum for project management with emphasis on less team meetings and more asynchronous communication through Google documents and Slack.

We work on new ideas in following 3 steps:

Ideation

In this step the team creates a list of ideas. Any member can take ownership of an idea and design an experiment to test it.

Refinement

After the Ideation step, the experiment owner refines the idea into functional specs and collaborates with other team members to deliver a new experiment. In this step, the experiment owner runs the experiment on users and ends the experiment after gathering sufficient data to calculate the experiment results.

Experimentation & Inference

Finally once the Refinement is done, the experiment owner has ended the experiment. The experiment owner then calculates the experiment result based on predefined metrics and determines next steps. The experiment owner also infers the experiment results and these insights would be used in future ideation.

Ideation is done with all the team members together while refinement and experimentation & Inference for each idea are done in parallel. We repeat all these steps after every 3 months.

We briefly described how we work as a team. Let us take a detailed look at each step.

Ideation: How do we come up with ideas ?

Ideas are based on hypotheses. Hypotheses are like foundational blocks for features, as you need a good foundation to build a tall building, similarly you need good hypotheses to build new features. The hypothesis also makes sure we focus on solving the biggest problems for our customers. We create good hypotheses through methods like user surveys, past experiments insight, data analysis etc.

After creating some hypotheses, we organize a team meeting to come up with new ideas. The meeting is contains 3 sections:

Kick-off , Inspiration and Ideation.

The first 2 sections provide valuable information to help team members come up with ideas in the Ideation section. The details about each section of the meeting are as following:

• Kick-off: In this section, we explain the team mission and discuss short term v/s long term goals for our team. We also explain the hypotheses we would be working on this section.
• Inspiration: In this section, we discuss past experiments and case studies to give concrete examples of how to design an experiment for a given idea.
• Ideation: We use the information provided in the Kick-off and Inspiration sections to perform 3 exercises to come up with new ideas: Sketch, Decide and Present.

For the ideation section, we perform following 3 exercises to come up with new ideas:
Note: In the first 2 exercises(Sketch and Decide), we split the team in groups of 4-5 members and each group performs the exercises separately. For the last exercise(Present), we perform it with the whole team.

Sketch

The aim of this exercise is to generate and share a broad range of ideas as individuals. Each member works individually and comes up with 1 idea for the next exercise. We work individually to prevent biases and groupthink. All the steps of this exercise are as following:

• Prepare: Take some time to prepare yourself for sketching ideas.
• Crazy 8s: Sketch 8 ideas in 8 minutes.
• Sketch your idea: Focus on 1 idea and elaborate it using three panel sketch
  

Decide

The aim of this exercise is to decide on the best ideas to take forward from each group. We assign one member from each group as the facilitator of the group. The facilitator decides which ideas would be presented to the whole team in the next exercise. All the steps for this exercise are as following:

• Silent review and dot vote: Individually review each sketch from the group and vote of specific areas from the sketch for further discussion.
• Group discussion: Discuss each sketch and clarify doubts.
• Final Vote: Vote on favorite idea for the next exercise.
• Final Decision: Facilitator picks 1~2 ideas from the group to present to the whole team in the next exercise.
  

Present

In this exercise, each group presents 1~2 ideas to the whole team. We follow these rules while presenting each idea:

• The author of the idea presents it to the whole team.
• The author provides all the necessary points discussed among the group for this idea in the previous exercise.
• The author states the hypothesis for the idea and explains how the idea is related to the hypothesis.
• We avoid any future discussion about the idea and try to understand each idea presented in the team.

After this exercise is done, we have a list of ideas that we can work on to deliver new experiments. Let’s discuss in the next section about refining these ideas into experiments.

Refinement: How do we refine ideas into experiments ?

Experiment owner refines an idea with the help of other members, especially product managers. Experiment owner can work on any of the following step from start till end of the experiment:

• Writing Project document
  • Doing research and analysis, stating the hypothesis, developing potential solutions
• Discussing experiment specs with team members
• Writing Experiment design document
  • Defining metrics and release case scenarios.
• Discussing with designers about ideal designs
• Clarifying specs
• Creating tickets
• Analyzing results of the experiment
• Determining next steps after experiment results

Taking ownership of an experiment is completely optional. The Product Manager supports experiment owners at each step and the experiment owner can flexibly discuss how much they want to contribute.
For example: Experiment owners can work on writing the project document and clarify the specs while product manager can work on other refinement steps

In the above section, we explained all the refinement steps for an experiment. Following are some of the team processes that helps refinement of ideas into experiments:

• Weekly team Sync meetings: This process helps in discussion/clarification of specs.
• Documentation and Templating: This process helps in writing project documents, experiment design documents, creating tickets and analyzing experiment results.
• Optional Team Processes: These are optional processes that help in speeding up the refinement process.

Weekly team Sync meeting

Experiment owners can discuss and clarify experiment specifications during this meeting. It provides an easy way to get feedback from the whole team.
Some of the rules that we follow for this meeting:

• All the discussion in this meeting are documented in a google document.
• Members need to add their discussion points before the meeting starts.
• In the meeting, all the members silently review all the discussion points and write questions or comments for each discussion point.
• If the discussion is taking a long time, we stop the discussion and continue it on slack or schedule a meeting.

Documentation and Templating

Documentation is very important to onboard a new experiment owner. We try to document every refinement step and provide references for a deeper understanding. When you are creating documentation for idea refinement, please keep following things in mind:

• Organize all the documents in one place.
• Document common practices you follow in your team for refining ideas.
• Provide references whenever possible for deeper understanding.

Templating is useful when we follow a similar structure for a refinement step. In our team, we have templates for following steps:

• Project document template: The template provides all the sections we need to write. It also provides references to help experiment owners write all the sections.
• Experiment design document template: We suggest common metrics and action plans that we follow in most of our experiments. The template also provides references to other documents that explains how to write your own metric and decide suitable action plans.
• Experiment analysis template: The template provides analysis for standard metrics that we use in our team. An experiment owner can modify the template based on their experiment metrics and evaluate the experiment result.
• JIRA Ticket template: We use JIRA for task management. We have created a document to note all the necessary tickets needed for an experiment and we have templates for creating individual tickets.

Optional Team Processes

Following are some of the other processes that we follow that may be useful for other teams in some cases:

• Create dedicated slack channels for each experiment.
  

  We have a large team and to reduce communication noise we create a slack channel for each experiment. The slack channel only contains those members that are working on the experiment.

• Create a kickoff meeting for each idea when we start working on it.
  

  Some ideas take a long time to clarify all the specs. Kickoff meetings help to clarify specs swiftly. These meetings are completely optional except for members you want to work on the idea.

Experimentation & Inference: How do we evaluate an experiment?

After an experiment ends, it is important to know what we learnt from the experiment. We keep following things in mind when we are concluding an experiment:

• We decide the metric and future action plan before we start an experiment. This is to avoid any bias we could get based on experiment results. You should not modify metrics and action plans after looking at the experiment results.
• We also note down what we learnt from the experiment. This is very helpful for conducting future experiments. Whatever are the experiment results please infer the result and share with all the team members.

Let’s take a look at the following hypothetical experiment to understand the above 2 points more clearly:
Experiment Details

Experiment: Send weekly notification to users at 4pm, Monday to inform them about the latest trending items on Mercari.

Hypothesis: Users will list more items if they are confident that their items would sell fast.

Goal Metrics: Number of listings per user
Action Plan:

• If the goal metric shows more than 2% lift, release the feature to all users.
• If the goal metric shows less than 2% lift, do not release the feature.

Experiment Result

• Goal metric shows 0% lift from the experiment.
• More than 90% of the users who got the notification, clicked on the notification to find more details.

Based on the experiment result, we stop sending the notification as we have decided earlier in our action plan. Although based on our analysis, we found out that users are interested in receiving trend related information as more than 90% of users clicked on the notification. Based on further analysis we found that:

• Most of the users do not list items at 4pm, Monday.
• Most of the users list items during weekend.

From this analysis we can infer following things:

• The time for sending a new notification should not be arbitrary as it could affect the outcome of the experiment.
• If the intent of a notification is to suggest a user to perform a certain action in the app, it would be better to send the notification when the user usually performs that action in the app.

These inferences could help us greatly in designing new experiments for sending similar notifications in future.

Sometimes teams do not find inferences from their experiments as finding inferences involve further analysis and research. The extra effort in finding inferences is very important as it greatly helps future experiments.

Summary

In this article we have explained in detail about how we work as a cross functional product team. Cross functional teams are made of people with different expertise to work together to deliver greater value to users. All the methods explained in this article helps empower all the members to achieve optimal results.

Empowering all the members to take ownership of new experiments is a very good strategy to make better products. Empowered Engineers FAQ article is very helpful in explaining many doubts in this area.