Почему утечки памяти в Python?



@zzmaster

У меня питон вылетает после 600 вставок записи в кликхаус.

tracemalloc.start()
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
snapshot1 = tracemalloc.take_snapshot()

    for row in rows:
        values = list(row.values()) # 94
        keys = list(row.keys()) # 95
        ret = client.insert(table, [values], column_names=keys) # 96
    
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
for stat in top_stats[:10]:
    print(stat)

tracemalloc после 100 проходов цикла распечатал следующее.

/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/context.py:24: size=25.8 KiB +25.8 KiB), count=171 (+171), average=155 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/client.py:651: size=15.9 KiB +15.9 KiB), count=115 (+115), average=142 B
/root/sync/dbsync.py:95: size=14.2 KiB (+14.2 KiB), count=114 (+114), average=127 B
/root/sync/dbsync.py:94: size=14.2 KiB (+14.2 KiB), count=114 (+114), average=127 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/query.py:293: size=4416 B (+416 B), count=46 (+46), average=96 B
/root/sync/dbsync.py:96: size=4176 B (+4176 B), count=115 (+115), average=36 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/insert.py:87: size=3648 B (+348 B), count=57 (+57), average=64 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/context.py:42: size=3648 B (+648 B), count=57 (+57), average=64 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/context.py:41: size=3648 B (+648 B), count=57 (+57), average=64 B
/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/client.py:656: size=3248 B (+248 B), count=58 (+58), average=56 B

Я проставил коментами номера строк файла dbsync.py

Отдельный вопрос, который, вероятно надо задавать не здесь — почему утечки в модуле clickhouse_connect, но почему также утечки в моем коде строки 94 95 96?

Сообщение об ошибке

Error during insert: :HTTPDriver for http://localhost:8123 returned response code 500)
 Code: 241. DB::Exception: Memory limit (total) exceeded: would use 1.01 GiB (attempt to allocate chunk of 5039783 bytes), maximum: 861.77 MiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker. (MEMORY_LIMIT_EXCEEDED) (version 23.11.2.11 (official build))

Traceback (most recent call last):
  File "/root/sync/dbsync.py", line 97, in <module>
    ret = client.insert(table, [values], column_names=keys)
  File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/client.py", line 554, in insert
    return self.data_insert(context)
  File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/httpclient.py", line 250, in data_insert
    response = self._raw_request(block_gen, params, headers, error_handler=error_handler, server_wait=False)
  File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/httpclient.py", line 431, in _raw_request
    error_handler(response)
  File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/httpclient.py", line 236, in error_handler
    self._error_handler(resp)
  File "/usr/local/lib/python3.10/dist-packages/clickhouse_connect/driver/httpclient.py", line 357, in _error_handler
    raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
clickhouse_connect.driver.exceptions.DatabaseError: :HTTPDriver for http://localhost:8123 returned response code 500)
 Code: 241. DB::Exception: Memory limit (total) exceeded: would use 1.01 GiB (attempt to allocate chunk of 5039783 bytes), maximum: 861.77 MiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker. (MEMORY_LIMIT_EXCEEDED) (version 23.11.2.11 (official build))


Решения вопроса 0


Ответы на вопрос 0

Добавить комментарий

Ваш адрес email не будет опубликован. Обязательные поля помечены *