Confidence by Spotify

🥁 We’re introducing Confidence Loop: a system that turns signals from real users into product understanding and improvements and, over time, allows more and more of those improvements to happen automatically. It connects related signals, surfaces recurring issues, and investigates what should change, keeping the evidence behind each issue. You choose how much of the loop runs autonomously, from surfacing issues to investigating and building fixes with human review. When human judgment is needed, the loop brings the problem, context, and possible paths forward, rather than leaving your team to assemble everything from scratch. Confidence Loop is now in early preview with design partners. Request access to join our design partner program and help shape what comes next. 👉 https://lnkd.in/exVu-Q2y

Your team is already talking about the product in Slack. Bring the evidence into the conversation. Mention @Confidence, ask your question, and follow up without repeating the context. You can also create flags and prepare experiments, right in the thread. Spend less time gathering findings across tools, and more time deciding what to do next, together. https://lnkd.in/dKGeWD4i

Your first experiment shouldn’t have to start with a data warehouse integration project. You have an idea to test. But your team doesn’t have a warehouse, or getting access to one means waiting for help from another team. We're now introducing Confidence Cloud: start experimenting without first setting up or connecting a data warehouse of your own. Confidence creates and manages the warehouse for you. Connect your application, send events, and create metrics to measure whether your product changes make a difference. Already have an established data warehouse? Well, the warehouse-native Confidence capabilities isn’t going anywhere. You can continue using your existing business data and custom SQL, with the flexibility and control of your own infrastructure. Same Confidence platform. Just two ways to get started. We’ve introduced several new capabilities in beta over the past few weeks, and we have more to come! Confidence Cloud adds an easier starting point, so more teams can spend less time on setup and more time building, measuring, and learning. https://lnkd.in/dKnZ8t64

Your dashboard answered the first question. What about the next five? We’ve recently introduced Confidence Agent and Session Replay in beta, and now we’re adding Product Analytics, connecting more user signals in Confidence. Ask questions in plain language and explore your warehouse data alongside your experiments, rollouts, and product context. “Where are users dropping off?”, “Which segments are affected?”, “What did we ship around the same time?”. Keep following the evidence, in Confidence or Slack, without building a new dashboard for every follow-up question or piecing together context across tools. And when you find a question worth tracking, turn it into a Pulse: a recurring update delivered to Slack that your team can reply to and explore further. Less time assembling the evidence. More time deciding what to investigate, test, and more importantly what to build next. Product Analytics is now in released in beta, sign up for free and try it out! https://lnkd.in/d8c4VvvP

The discussion around the "Why Spotify Doesn't Use Bayes" post continues! 💥🧠 We published our blog post and paper on Bayesian inference for A/B testing, and the responses have been fantastic. Michael Leznik wrote a reply that we found particularly thought-provoking, so we wrote a response and published it! I really appreciate this kind of nuanced, substance-over-punchlines discourse about methodology. Thanks, Michael! New response (with links to Michael's post): https://lnkd.in/ei8n8fjS Original post: https://lnkd.in/p/enEW-jRy

🎉 The Confidence, by Spotify crew (including me) will be in Boston for CODE @ MIT in November! Come have a drink with us at the Spotify office on the 12th! My good friend Hannes Lagerroth from Lovable will be joining us to talk about how they do experimentation. Space is limited, so grab a spot now! There will be beers and pizza, and an excellent opportunity to tell me what you really thought about our recent Bayes paper 😅 https://lnkd.in/dJwDb87p

Your experiment flagged a problem. Now you need to understand what happened. Introducing Session Replay in beta, in Confidence, bringing real user recordings together with your feature flags, experiments, and release history. And no, you don’t have to watch hours of recordings to find the moments that matter. The Confidence Agent that we released last week, will review sessions for you, surfacing bugs and opportunities to improve. You can jump straight to the relevant moment in a recording to see what users experienced. Imagine your checkout experiment shows rising cart abandonment. You ask the agent to investigate, and it identifies a field covering the checkout button on some mobile browsers. Now you have something concrete to fix and test, rather than another round of guesswork. That’s the value of connecting flags and experimentation with recordings: you can move from measuring an outcome to investigating the experience behind it, without piecing together context across tools. Recordings is now available to all Confidence users. Read the full feature reveal link in the comments 👇

Going to CODE @ MIT? Join us the night before. Thursday November 12, Spotify's Boston office. Hannes Lagerroth from Lovable shares how Lovable combines their own insights apps with Confidence to experiment across their stack. Pizza and drinks for everyone. Open to conference attendees and friends. Save your spot: https://lnkd.in/eQz3hV_V

Bayesian or frequentist? We hear this question constantly at Spotify. Mattias Frånberg and Mårten Schultzberg recently published a paper and blog post explaining why we don't think the framework choice matters as much as people believe. What matters more: is the inference coherent? Does the sample size calculator match how decision rules are actually applied? Are experiments producing learning, not just ship decisions? That's what we built Confidence around. https://lnkd.in/eGSrh6Kx

Introducing Confidence Agent, now in beta. It's an AI collaborator built into Confidence that works directly with your flags, experiments, metrics, and project documents. Describe what you want, and it helps shape the brief, configures the flag and the experiment, and explains what the results actually say. An experimentation platform exists to find the truth, not to confirm hypotheses. Confidence Agent holds the same line: underpowered experiment, sample-ratio mismatch, confidence interval too wide to decide, it says so instead of manufacturing a recommendation. It removes the configuration and translation work between steps so human judgment gets spent where it actually matters. Available to all workspaces today. Try it and tell us what you think: https://lnkd.in/eXpjDUaz