In this article we will be exploring the difference between Mixpanel and Amplitude funnel reports. Be advised that this article can be considered as an in-depth analysis rather than a light read.
Test infrastructure
For this test we are using Segment. The reason is that a single set of events is being recorded and forwarded to both Mixpanel and Amplitude. As a result any deviations in results can be attributed to the inner workings of each tool rather than differences in how events were being recorded.
The following events were sent to Mixpanel and Amplitude via Segment.
Understanding totals
Mixpanel and Amplitude funnels allow for the option to build funnels using “uniques” or “totals”. Definitions for Amplitude can be found here and for Mixpanel here.
After reading the documentation the conclusion should be that they represent the same thing regardless of the platform. But do they really show the same outcome when used in funnels?
Let’s test it out.
Baseline testing
We are sending 1 event twice – Funnel Event 1 Finalised – and 1 event once – Funnel Event 2 Finalised.
Timestamps are highlighted to indicate a difference of more than 10 seconds between the 1st and 2nd events (Funnel Event 1 Finalised and Funnel Event 1 Finalised) and the 2nd and 3rd event (Funnel Event 1 Finalised and Funnel Event 2 Finalised).
We’ll use the same configuration for both reports to start with.
Funnel start = Funnel Step 1 Finalised
Funnel end = Funnel step 2 Finalised
Count as totals
Set a conversion window of 1 hour – This is way beyond the time between the ingestion of the very first event and the last event ingested
Even though the configuration is the same. The results look different:
Amplitude indicates a total count of 2 for the first step of the funnel.
Amplitude indicates a conversion rate of 50%.
Mixpanel indicates a total count of 1 for the first step of the funnel.
Mixpanel indicates a conversion rate of 100%.
Conversion window tweaks
If we lower the conversion window to 10 seconds the results are the following:
Results for Amplitude remain stable.
Results for Mixpanel however change and now replicate the results from Amplitude (with the current and previous 1 hour conversion window.
Mixpanel indicates a total count of 2 for the first step of the funnel.
Mixpanel indicates a conversion rate of 50%.
The reason of this change is the following:
In Mixpanel we are essentially asking “how many times was a funnel completed” when using totals (I quoted this from a Mixpanel support ticket FYI). The user can not re-enter the funnel until the user converts or the conversion window runs out. While in Amplitude a user can re-enter.
In our first example the conversion window was set to 1 hour. The second funnel entry was therefore discarded since the user did not complete the funnel nor did the conversion window of 1 hour expire. As a result we achieve 100% conversion rate.
In our second example we lowered the conversion window to 10 seconds. The time between the first funnel entry and the second was more than 10 seconds. Therefore the second funnel entry was counted and conversion rate dropped from 100% (1 hour conversion window) to 50% (10 second conversion window).
Amplitude however answers the question “how many funnel entries were completed” when using totals. Meaning Amplitude counts every funnel entry separately regardless of the conversion window set. This leads to consistent answers in both conversion windows.
Edge case testing with the conversion window
“The second funnel entry was therefore discarded since the user did not complete the funnel nor did the conversion window expire.” – This triggered questions and invited us to run an additional test to evaluate the behavior of Mixpanel funnels.
What happens if an event was triggered twice within the conversion window, and therefore not counted (according to Mixpanel’s logic), but the conversion actually happened outside of the conversion window compared to the first event, but within the conversion window for the second event?
To verify this edge case we triggered 3 events each +/- 7 seconds apart.
In Amplitude we get the following results in a funnel reports using totals and a 10 second conversion window:
Amplitude answers the question “how many funnel entries were completed” and answers 50% which is correct versus the events we have sent. This is consistent with our previous findings.
For the same configuration in Mixpanel however we get different results.
Here the conversion rate is 0%. Reason being that the duplicate event was still in the conversion window and therefore dropped. Comparing the first event with the Funnel Event 4 Finalised event has a conversion window greater than 10 seconds. Therefore no conversion is measured.
This seems to be a false negative due to the fact that Mixpanel drops the second (duplicate event) since in practice the answer to the question “how many times was a funnel completed (within 10 seconds)” is 1.
If we expand the conversion window to 20 seconds we see the conversion show up as in our previous example using totals.
A word on uniques
Compared to totals, uniques will not allow a user to re-enter a funnel. Not in Mixpanel, nor in Amplitude. Uniques answers the question “how many users were able to complete the funnel in conversion window X?”.
Mixpanel is quite strict, as we have seen with totals, on the way that only the first funnel entry is being used.
Let’s use the same event series to evaluate how uniques affects the counts.In Mixpanel the result of a 10 second conversion window is the following:
Mixpanel indicates a total count of 1 for the first step of the funnel.
Mixpanel indicates a conversion rate of 0%.
If we try to answer the question “how many users were able to complete the funnel in conversion window of 10 seconds” the answer is 0 according to Mixpanel. Even though our test actually shows that we were able to complete it within 10 seconds. This false negative is due to the fact that Mixpanel drops the duplicate event for funnel entries as it only looks at the earliest entry.
Amplitude however shows a different story:
Amplitude indicates a total count of 1 for the first step of the funnel.
Amplitude indicates a conversion rate of 100%.
This is correct.
Even though by default Amplitude also looks at the first event (vs any following duplicates) for funnel entries, Amplitude has a built-in failsafe logic that will look at the next duplicate if the first event falls out of the conversion window. This compensates for the edge case we are investigating here and allows for a correct answer to be formulated.
