I’ve recently run into an issue where a data-processing task in a Symfony app would crash with a PHP ‘Out of memory’ fatal error. The task in question was using Doctrine to retrieve some rows from the database, updating some entities, creating some others and saving the changes back to the database. All was well initially when testing the process with a couple of hundred rows, but as the amount of data being processed increased to a few thousand rows and then tens of thousands, fatal errors began to pop up.
At first I assumed that it was simply a case of too much data being fetched from the database at once and that processing the data in smaller batches would avoid the errors. Unfortunately, after trying just that and still running into the dreaded out of memory errors, it seemed that something else was afoot.
Sparing you the details of the gnashing of teeth and pounding of fists that took place between the realisation above and eventually locating the source: it turned out that the problem was my Symfony app’s Monolog configuration. I’d been using a Monolog configuration very similar to this Symfony default:
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
handler: nested
nested:
type: stream
path: "%kernel.logs_dir%/%kernel.environment%.log"
level: debug
Note the ‘fingers_crossed’ log handler, if its action level is not hit, it will happily buffer all the log messages sent to it in case they are needed at some point in the future. In my app’s case, between Doctrine and my own debug messages, what I think could technically be described as a ton of log messages below the handler’s action level were being buffered for later use until the buffer became so big that PHP used up as much memory as it was allowed and failed with an out of memory error.
The solution was depressingly simple: add a ‘buffer_size’ parameter to the fingers_crossed handler. This limits the number of messages the handler will buffer, and in a long running script that doesn’t often hit the action level can dramatically reduce the amount of memory used by Monolog.
A fixed config would look something like the one shown below, note the ‘buffer_size’ in the main handler.
monolog:
handlers:
main:
type: fingers_crossed
action_level: error
buffer_size: 200
handler: nested
nested:
type: stream
path: "%kernel.logs_dir%/%kernel.environment%.log"
level: debug
The ‘buffer_size’ parameter is listed in Symfony’s Monolog Configuration Reference, but somewhat annoyingly what it actually does isn’t mentioned.
Read more of my posts on my blog at http://blog.tinned-software.net/.
