PHP Meminfo Treemap

I ran across a really neat PHP extension called php-meminfo while debugging an out-of-memory error at work. It adds a function that you can call from PHP which will walk the call stack and recursively extract every declared variable to produce a heap dump. All of the information about each variable and each of its children (for arrays and objects) is written to a file in JSON format. The extension also includes some CLI analyzer tools to query the heap dump in different ways.

I thought that it would be really cool to visualize this data as a tree map. At the time, I didn't even know the name for this kind of visualization. I just remembered it from using WinDirStat, a handy Windows tool for visualizing disk space usage. I was imagining the same, but for the PHP heap. Personally, I think this kind of visualization really makes certain issues pop out right away. Also, it's just really fun to to explore!

The first task for producing this visualization is to process a (usually gigantic) JSON heap dump. I chose Rust for this job in part because it's a good fit for the task, and also because I wanted an excuse to learn more. We can think of the heap as a big graph of objects that probably has lots of cycles. In the JSON format, all of these objects have been flattened out and given an ID. Items like arrays and objects have a list of their children's IDs.

{
  "items": {
    "0x7fe7d24ec130" : {
        "type" : "array",
        "size" : "72",
        "symbol_name" : "error",
        "is_root" : true,
        "frame" : "oom_heap_dump_shutdown()",
        "children" : {
            "type":"0x7fe7d2030340",
            "message":"0x7fe7d2030360",
            "file":"0x7fe7d2030380",
            "line":"0x7fe7d20303a0"
        }
    },
    "0x7fe7d2030340" : {
        "type" : "int",
        "size" : "16",
        "is_root" : false
    },
    // ...
}

Implementation Details🔗

We start by loading all of the items into a BTreeMap (serde makes this super easy). Now the problem is that we've got a graph of items from the heap, but to make a treemap, we need to lower the graph representation into a tree somehow. My solution to this is to make the user choose a root item. Then we can "hang" the graph from this root node and prune away cycles. This involves a graph traversal starting with the root and building a tree. When we see a node we've already visited, we don't add it to our tree again. This works surprisingly well under some circumstances. The caveat is that even if the user chooses a good root, we might still be missing a lot of data from our tree. The heap graph often has multiple components (groups of items that are not connected to the rest of the graph). In the future, I would like to explore some process for identifying all of the components in the heap graph and then devising some hueristic to make a treemap which includes all items.

Once we've converted the heap into a tree, we can aggregate bottom up to get the total size below each node in the tree. Since the tree was created from a graph, the order that items were visited by the traversal can have a big impact on the total size of their sub-tree. Basically, the processing of going from graph -> tree could produce many, many different trees so you could say that an items parent in the tree is not so stable, and also that our visualization might not produce exactly what you expect for a given heap dump.

The program offers multiple output options, but the html option is the most useful. It performs a breadth-first traversal from the top and outputs a flat list of nodes with their aggregated sizes into JSON that can be used by google charts treemap. This provides a nice looking interactive treemap that the user can dig into. The limits are important to keep things performant, so after the limit is reached, all nodes further down in the tree will get bundled up into an "other" category.

Getting back to my original memory issues at work. While the treemap visualization of the heap did make the problem super obvious (it works!), I had already found the issue in the meantime, before php-meminfo-treemap was ready. It turned out the job that was running out of memory was iterating through data from all clients across all DB shards (which is pretty unusual for a job), and the culprit was that all feature flag values for each client were loaded into memory and not released. Each feature flag was a class with a bunch of junk, so after processing thousands of clients that each cause thousands of feature flag instances to be loaded we ended up with millions of these feature flag instances hogging all of the memory.

This project was a ton of fun to work on and I learned so many new things! It's not very refined and I don't think there's a ton of demand for a tool like this, but I hope I can find some excuse to revisit it in the future to improve it.