Python, json and garbage collection

We have a webapp which exposes a REST interface. Json is used as the data format for most of the apis.  98% of those json messages were less than 500 Kb. But 2% of them can go above 100 MB. It was observed that after processing one such 100 MB json message, the process memory went up to 500 MB and stayed there. It never came down even after running the webapp for hours and processing small json messages. Interesting observation is that memory remained at 500 MB even after processing multiple 100 MB json messages. On analyzing the problem it was found that 'json.loads' is the culprit. Calling gc.collect does releases the memory. And for now that seems to be the only solution.

The memory is not held up in any caches or python's internal memory allocator as the explicit call to gc.collect is releasing memory. It seems the gc threshold was never reached and as a result garbage collection never kicked in. But it seems strange that threshold was never reached even after running the webapp for hours.

Test code

The test code shown below simulates the situation .If the call to json.loads is omitted then this issue is not observed.  GC counts printed after invocation of jsontest is (150, 9, 0)  which indicates that gc threshold (700, 10, 10) is not met.


import json
import time
import gc
from itertools import count

gc.set_debug(gc.DEBUG_STATS)

def keygen(size):
    for i in count(1):
        s = str(i)
        yield '0' * (size - len(s)) + str(s)

def jsontest(num):
    keys = keygen(20)
    kvjson = json.dumps(dict((keys.next(), '0' * 200) for i in range(num)))
    kvpairs = json.loads(kvjson)
    del kvpairs # Not required. Just to see if it makes any difference                            
    print 'load completed'

jsontest(500000)
print gc.get_count()

while 1:
    time.sleep(1)

Comments

Post a Comment

Popular posts from this blog

select, poll and epoll, a twisted story

Some time back I had to write a network server which need to support ~50K concurrent clients in a single box. Server-Client communication used a propitiatory protocol on top of TCP where JSON/NetString is used as the messaging format. Clients exchanged periodic keep-alives which server used to check health state. Server also exposed a REST interface for other modules to communicate with client.  As most of the operations were IO based(socket/db) we decided to used python/twisted to implement server. On performing load tests we found that server is able to handle only 1024 client after which connections are failing. Increased per process max open files (1024) to 100000 (ulimit -n 100000) and still the connections failed at 1024. select limitation select fails after 1024 fds as  FD_SETSIZE max to 1024. Refer to  blog post  for details. Twisted's default reactor seems to be based on select. As a natural progression  poll  was tried next to over come max open fd issue. poll limi
Read more

Python, Ruby and OOPs

I have found quite some blogs and articles mentioning python as not being completly object oriented and ruby as a better alternative in this perspective. Of course all entities in python are objects. But instead of providing methods to objects to solve all problems, python provides a mixture of  builtin functions , module functions and object methods. I like python's approach and feel that is the right one. Can all problems be modeled and solved elegantly using object and methods ? I don't think so. Following is from an  interview  with Alexander Stepanov the creator of C++ STL where he mentions. I find OOP methodologically wrong. It starts with classes. It is as if mathematicians would start with  axioms. You  do not start with axioms - you start with proofs. Only when you have found a bunch of related proofs, can you come up with axioms. You end with axioms. The same thing is true in programming: you have to start with interesting algorithms. Only when you understand them
Read more