The are we slim yet (AWSY) benchmark measures memory usage. Recently when I made a simple change to firefox and expected it might save a bit of memory, it actually increased memory usage on the AWSY benchmark.
We have lots of tools to hunt down memory usage problems. But to see an almost "log" of when garbage collection and cycle collection occurs, the Firefox profiler is amazing.
I wanted to profile the AWSY benchmark to try and understand what was happening with GC scheduling. But it didn’t work out-of-the-box. This is one of those blog posts that I’m writing down so next time this happens, to me or anyone else, although I am selfish. And I websearch for "AWSY and Firefox Profiler" I want this to be the number 1 result and help me (or someone else) out.
The normal instructions
First you need a build with profiling enabled. Put this in your mozconfig
ac_add_options --enable-debug ac_add_options --enable-debug-symbols ac_add_options --enable-optimize ac_add_options --enable-profiling
The instructions to get the profiler to run came from Ted Campbell. Thanks Ted.
Ted’s instructions disabled stack sampling, we didn’t care about that since the data we need comes from profile markers. I can also run a reduced awsy test because 10 entries is enough to create the problem.
export MOZ_PROFILER_STARTUP=1 export MOZ_PROFILER_SHUTDOWN=awsy-profile.json export MOZ_PROFILER_STARTUP_FEATURES="nostacksampling" ./mach awsy-test --tp6 --headless --iterations 1 --entities 10
But it crashes due to Bug 1710408.
So I can’t use nostacksampling, which would have been nice to save some
memory/disk space, never mind.
So I removed that option, then I get profiles that are too short. The profiler records into a circular buffer so if that buffer is too small it’ll discard the earlier information. In this case I want the earlier information because I think something at the beginning is the problem. So I need to add this to get a bigger buffer. The default is 4 million entries (32MB).
export MOZ_PROFILER_STARTUP_ENTRIES=$((200*1024*1024))
But now the profiles are too big and Firefox shutdown times out (over 70 seconds) so the marionette test driver kills Firefox before it can write out the profile.
The solution
So we hack
testing/marionette/client/marionette_driver/marionette.py
to replace shutdown_timeout with 300 in some places.
Setting DEFAULT_SHUTDOWN_TIMEOUT and also self.shutdown_timeout to 300
will do.
There’s probably a way to pass a parameter, but I didn’t find it yet.
So after making that change and running ./mach build the invocation is now:
export MOZ_PROFILER_STARTUP=1 export MOZ_PROFILER_SHUTDOWN=awsy-profile.json export MOZ_PROFILER_STARTUP_FEATURES="" export MOZ_PROFILER_STARTUP_ENTRIES=$((200*1024*1024)) ./mach awsy-test --tp6 --headless --iterations 1 --entities 10
And it writes a awsy-profile.json into the root directory of the project).
Hurray!
Follow-up
Whimboo says that setting toolkit.asyncshutdown.crash_timeout might help.
But it may wait until after some stuff has been implemented:
a solution here should also be to extend the toolkit.asyncshutdown.crash_timeout value
oh wait. actually we haven’t fixed that yet, but only use it via geckodriver`