article thumbnail

How to Segment A/B Test Results to Find Gold

ConversionXL

This is especially true of big businesses: Chad Sanderson, Subway. In any experiment, some metric within some group usually changes, but whether we’ve invested in the statistical rigor to discover it is another question altogether.”. Chad Sanderson explains that you must have a methodology to segmentation: Chad Sanderson, Subway.

Chad 112
article thumbnail

Lessons Learned: Continuous deployment and continuous learning

Startup Lessons Learned

Our tests suite takes nine minutes to run (distributed across 30-40 machines). Luckily, Chad Austin has recently weighed in with an excellent piece called 10 Pitfalls of Dirty Code. The high level of our process is dead simple: Continuously integrate (commit early and often). On commit automatically run all tests.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Lessons Learned: What does a startup CTO actually do?

Startup Lessons Learned

October 2, 2008 8:58 AM Chad said. Startup Visa update ► February (5) Kiwi lean startup + Australia next Why diversity matters (the meritocracy business) Beware of Vanity Metrics (for Harvard Business Rev. Well it will be my job description once my team expands some more and I can delegate some of my daily engineering tasks.

CTO 168
article thumbnail

CXL Live 2018 Recap: Top 5 Lessons from Each Speaker

ConversionXL

Work hard to define meaningful product metrics – enabler of team success. Chad Sanderson – The Statistical Pitfalls of A/B Testing. In an A/A test the distribution of p-values is random. There’s no single metric to track, there are many: Leads. Iterate to the vision. Have a vision. . Quality of leads.

Retention 106
article thumbnail

How to Structure Your Optimization and Experimentation Teams

ConversionXL

In decentralized (or “embedded”) teams, optimization responsibilities are distributed amongst employees under many different umbrellas within the organization, rather than one department. Chad Sanderson, the Experience Optimization Manager for Subway, creates an analogy for how the centralized model is set up: Chad Sanderson, Subway.

Chad 90
article thumbnail

CXL Live 2019 Recap: Takeaways from Every Speaker

ConversionXL

There are tons of metrics you can monitor: Conversion rate; Bounce rate ; Click-through rate ; Pageviews; Lead captures; Purchase conversion rate. Setting up your experiments for success: Define success metrics. Statistical ghost: When you think your test has impacted your metric, but in reality there’s no impact.