2026.06.04
Спасибо, что уточнили — «allocation stall» в логах ZGC очень многое объясняет.
Что значит «allocation stall» у ZGC?
Это не OOM, а предсмертное состояние.
ZGC не успевает освобождать память так же быстро, как приложение её выделяет. Когда куча заполнена на критический уровень (High Threshold), ZGC перестаёт принимать новые аллокации и заставляет все треды приложения ждать (stall), пока не завершится цикл сборки.
Если это длится долго — JVM выбрасывает OOM, хотя технически память ещё может быть не до конца исчерпана.
Типичные причины для вашей конфигурации (26 ГБ, ZGC)
1. Недостаточно памяти для ZGC
ZGC требует память под вспомогательные структуры (например, ZPage Cache). Эффективный размер кучи может быть на 10–15% меньше заданных 26 ГБ.
Решение: уменьшить -Xmx до 22–23 ГБ, освободив системе буфер.
2. Слишком быстрый аллокационный rate (особенно при потоковом импорте или массовой обработке через Kafka)
ZGC умеет держать паузы <1 мс, но пропускная способность на сборку — не бесконечная. При аллокациях >2–4 ГБ/сек начинаются сталы.
Диагностика: замерить через -Xlog:gc+stats=debug или JMX метрику ZGC%.
3. Утечка памяти внутри IDM (например, ConcurrentHashMap без эвикшена, статические кэши)
Проверьте в логах до падения: если сталы возникали периодически и нарастала базовая занятость кучи после GC (Used не опускается ниже N%) — это утечка.
4. OpenSearch + Kafka на том же стенде
Даже 47 ГБ суммарно — тесно. Если OS или Kafka делают page cache thrashing, JVM IDM может страдать от внезапных задержек ввода-вывода, из-за чего ZGC не успевает выполнять concurrent фазы.
Рекомендация: выделить для IDM изолированные CPU ядра (taskset) и поднять приоритет GC тредов.
Как точно определить причину по логам
Найдите в gc.log строки непосредственно перед OOM. Примерно так:
```
[ZGC] Allocation stall (creeping...)
[ZGC] Allocation stall (0.512ms) # если сталы в миллисекундах — OK
[ZGC] Allocation stall (1523ms) # если больше 200 мс — плохо
```
Ищите:
· Allocation stall с длительностью >500 мс
· Фразу Concurrent Mark aborted или OutOfMemory
· Статистику перед крахом: Used: 25.8G, High: 26G, Mark start: 25.1G
Немедленные действия (пока без правок кода)
Временный фикс (если нет утечки):
```bash
-Xmx22G -Xms22G -XX:+UseZGC -XX:ConcGCThreads=8 -XX:ZAllocationSpikeTolerance=5
```
· ZAllocationSpikeTolerance=5 — разрешает ZGC чуть агрессивнее запускать сборки при всплесках.
Если утечка есть — без дампа кучи (jmap -dump:live) не обойтись.
Вы можете прислать последние 50–100 строк из gc.log до OOM (можно вырезку). Я расшифрую, что именно происходит: утечка, недостаточный размер кучи или задержки от соседей.
Это важный симптом. ZGC простаивает, а память переполняется — значит, узкое место не в CPU, а в аллокациях (выделении памяти) или паузах на ввод-вывод.
Вот что обычно происходит в вашей ситуации:
1. ZGC не успевает за аллокациями, но CPU не загружен
ZGC работает конкурентно (вместе с тредами приложения). Если приложение выделяет объекты быстрее, чем ZGC успевает их чистить, то:
· Растёт занятость кучи → аллокационные сталы → OOM.
· Но CPU может быть низким, потому что ZGC не требует много ядер (обычно 1–2 на 100 ГБ кучи), а приложение сидит в ожидании памяти (stall).
2. Самая частая причина на вашем стенде: Kafka или OpenSearch забирают I/O
IDM, скорее всего, пишет/читает топики или индексирует данные через OpenSearch. Если диски медленные или перегружены:
· ZGC пытается сделать mmap/unmap страниц кучи (ZGC использует виртуальную память).
· I/O-задержки растягивают фазы сборки → аллокации обгоняют GC → сталы.
· При этом CPU потребляется мало, потому что время уходит на ожидание диска.
3. Ещё вариант: слишком большой -Xmx для доступной физической памяти
Если 26 ГБ + OpenSearch + Kafka дают подсистеме памяти трепетать (swapping, thrashing page cache), то ZGC может простаивать в ожидании страниц. Проверьте free -h, нет ли swap и высокого si/so в vmstat 1.
Что делать прямо сейчас
1. Быстрая диагностика простоем CPU:
```bash
# Посмотреть, не ждёт ли IDM ввода-вывода
pidstat -d 1 -p $(pgrep -f idm)
# Если %wait высокий (>10%) — диск тормозит
```
2. Проверьте, нет ли длительных аллокационных задержек (из gc.log):
```bash
grep "Allocation stall" gc.log | tail -20
```
Если задержки >500 мс — подтверждение.
3. Временное решение — уменьшить -Xmx до 20–22 ГБ (чтобы снизить нагрузку на I/O и page cache).
4. Постоянно — разнести по разным машинам OpenSearch и Kafka, либо хотя бы по разным дискам.
Пришлите вывод pidstat -d и vmstat 1 5 — скажу точно, диск ли виноват.
Спасибо за цифры. 272–756 КБ/с записи — это очень мало, почти простаивающий диск. Значит, диск не причина.
Ситуация становится яснее: CPU простаивает, диск не нагружен, память переполняется, а ZGC в сталлах.
Остаются два реальных варианта:
1. Приложение просто выделяет память быстрее, чем ZGC успевает чистить
Это случается при:
· Циклах с массовым созданием объектов (например, распарсили 10 ГБ JSON в HashMap)
· Потоковой обработке больших сообщений из Kafka без batch-коммитов
· При этом GC не нужен CPU для сборки, потому что фазы ZGC (маркировка, релокация) конкурентны, но их пропускная способность ограничена. Аллокатор просто обгоняет сборщик.
Решение:
· Добавьте флаг: -XX:ConcGCThreads=12 (у вас 20 ядер, можно дать ZGC больше потоков)
· Увеличьте -XX:ZCollectionInterval=30 (чтобы ZGC запускался чаще)
· Но главное — профилировать аллокации в IDM. Запустите с -XX:+HeapSampling или сделайте дамп при OOM.
2. Прерывистый характер работы (burst allocation)
Если IDM долго ждёт данные, потом получает гигантский batch из Kafka, выделяет под него память мгновенно, а ZGC не успевает активироваться.
Проверка: в логах IDM есть ли паузы по 5–10 минут, потом резкий всплеск?
Решение:
```bash
-XX:SoftMaxHeapSize=20G -XX:ZUncommitDelay=5000
```
Это заставит ZGC держать меньше памяти и чаще собирать.
Срочный вопрос к вам (решающий)
Чем именно IDM занимается при падении?
· Импорт большого файла?
· Массовая рассылка писем?
· Синхронизация с AD / OpenSearch?
И второе: покажите полные опции JVM для IDM (особенно -Xlog:gc* и настройки ZGC).
Скорее всего, проблема не в системе, а в паттерне аллокаций кода. Без этого придётся увеличивать кучу до 30–32 ГБ и надеяться, но это ненадёжно.
Что в логе:
026-06-05T08:20:13.098+0300][3571566][gc] GC(50) Concurrent Mark Cycle 7234,450ms
[2026-06-05T08:20:13.951+0300][3571566][gc] GC(57) Pause Young (Normal) (G1 Evacuation Pause) 24643M->24052M(26624M) 65,993ms
[2026-06-05T08:20:14.654+0300][3571566][gc] GC(58) Pause Young (Concurrent Start) (G1 Evacuation Pause) 25172M->24704M(26624M) 70,657ms
[2026-06-05T08:20:14.654+0300][3571566][gc] GC(59) Concurrent Mark Cycle
[2026-06-05T08:20:15.450+0300][3571566][gc] GC(60) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 25824M->26083M(26624M) 95,480ms
[2026-06-05T08:20:18.009+0300][3571566][gc] GC(61) Pause Full (G1 Compaction Pause) 26083M->25206M(26624M) 2558,703ms
[2026-06-05T08:20:18.010+0300][3571566][gc] GC(59) Concurrent Mark Cycle 3355,711ms
[2026-06-05T08:20:18.533+0300][3571566][gc] GC(62) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 26294M->26294M(26624M) 64,760ms
[2026-06-05T08:20:22.002+0300][3571566][gc] GC(63) Pause Full (G1 Compaction Pause) 26294M->25642M(26624M) 3468,206ms
[2026-06-05T08:20:22.392+0300][3571566][gc] GC(64) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 26346M->26346M(26624M) 50,191ms
[2026-06-05T08:20:22.392+0300][3571566][gc] GC(66) Concurrent Mark Cycle
[2026-06-05T08:20:25.716+0300][3571566][gc] GC(65) Pause Full (G1 Compaction Pause) 26346M->25913M(26624M) 3323,877ms
[2026-06-05T08:20:25.717+0300][3571566][gc] GC(66) Concurrent Mark Cycle 3325,625ms
[2026-06-05T08:20:25.973+0300][3571566][gc] GC(67) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 26329M->26329M(26624M) 32,570ms
[2026-06-05T08:20:28.511+0300][3571566][gc] GC(68) Pause Full (G1 Compaction Pause) 26329M->26096M(26624M) 2537,818ms
[2026-06-05T08:20:28.596+0300][3571566][gc] GC(69) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 26224M->26224M(26624M) 14,162ms
[2026-06-05T08:20:28.596+0300][3571566][gc] GC(71) Concurrent Mark Cycle
[2026-06-05T08:20:32.018+0300][3571566][gc] GC(70) Pause Full (G1 Compaction Pause) 26224M->26152M(26624M) 3421,468ms
[2026-06-05T08:20:32.018+0300][3571566][gc] GC(71) Concurrent Mark Cycle 3422,145ms
[2026-06-05T08:20:32.050+0300][3571566][gc] GC(72) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 26184M->26184M(26624M) 12,315ms
[2026-06-05T08:20:34.677+0300][3571566][gc] GC(73) Pause Full (G1 Compaction Pause) 26184M->26167M(26624M) 2626,999ms
[2026-06-05T08:20:35.061+0300][3571566][gc] GC(74) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 26615M->26615M(26624M) 33,773ms
[2026-06-05T08:20:35.061+0300][3571566][gc] GC(76) Concurrent Mark Cycle
[2026-06-05T08:20:37.791+0300][3571566][gc] GC(75) Pause Full (G1 Compaction Pause) 26615M->26368M(26624M) 2730,331ms
[2026-06-05T08:20:37.794+0300][3571566][gc] GC(76) Concurrent Mark Cycle 2732,829ms
[2026-06-05T08:20:37.934+0300][3571566][gc] GC(77) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 26592M->26592M(26624M) 18,255ms
[2026-06-05T08:20:41.416+0300][3571566][gc] GC(78) Pause Full (G1 Compaction Pause) 26592M->26465M(26624M) 3482,126ms
[2026-06-05T08:20:41.513+0300][3571566][gc] GC(79) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 26593M->26593M(26624M) 25,072ms
[2026-06-05T08:20:41.513+0300][3571566][gc] GC(81) Concurrent Mark Cycle
[2026-06-05T08:20:44.161+0300][3571566][gc] GC(80) Pause Full (G1 Compaction Pause) 26593M->26521M(26624M) 2647,631ms
[2026-06-05T08:20:44.163+0300][3571566][gc] GC(81) Concurrent Mark Cycle 2649,694ms
[2026-06-05T08:20:44.228+0300][3571566][gc] GC(82) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 26617M->26617M(26624M) 12,292ms
[2026-06-05T08:20:47.230+0300][3571566][gc] GC(83) Pause Full (G1 Compaction Pause) 26617M->26563M(26624M) 3001,912ms
[2026-06-05T08:20:47.258+0300][3571566][gc] GC(84) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 26595M->26595M(26624M) 9,735ms
[2026-06-05T08:20:47.258+0300][3571566][gc] GC(86) Concurrent Mark Cycle
[2026-06-05T08:20:49.867+0300][3571566][gc] GC(85) Pause Full (G1 Compaction Pause) 26595M->26576M(26624M) 2608,287ms
[2026-06-05T08:20:49.868+0300][3571566][gc] GC(86) Concurrent Mark Cycle 2609,472ms
[2026-06-05T08:20:49.911+0300][3571566][gc] GC(87) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 26608M->26608M(26624M) 20,726ms
[2026-06-05T08:20:53.359+0300][3571566][gc] GC(88) Pause Full (G1 Compaction Pause) 26608M->26593M(26624M) 3447,028ms
[2026-06-05T08:20:58.171+0300][3571566][gc] GC(89) Pause Full (G1 Compaction Pause) 26593M->26531M(26624M) 4812,235ms
[2026-06-05T08:20:58.218+0300][3571566][gc] GC(90) Pause Young (Concurrent Start) (GCLocker Initiated GC) (Evacuation Failure) 26595M->26595M(26624M) 12,023ms
[2026-06-05T08:20:58.218+0300][3571566][gc] GC(92) Concurrent Mark Cycle
[2026-06-05T08:21:00.831+0300][3571566][gc] GC(91) Pause Full (G1 Compaction Pause) 26595M->26556M(26624M) 2612,964ms
[2026-06-05T08:21:00.835+0300][3571566][gc] GC(92) Concurrent Mark Cycle 2616,554ms
[2026-06-05T08:21:00.861+0300][3571566][gc] GC(93) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 26588M->26588M(26624M) 11,201ms
[2026-06-05T08:21:04.001+0300][3571566][gc] GC(94) Pause Full (G1 Compaction Pause) 26588M->26569M(26624M) 3140,167ms
[2026-06-05T08:21:04.031+0300][3571566][gc] GC(95) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 26601M->26601M(26624M) 10,343ms
[2026-06-05T08:21:04.031+0300][3571566][gc] GC(97) Concurrent Mark Cycle
[2026-06-05T08:21:06.653+0300][3571566][gc] GC(96) Pause Full (G1 Compaction Pause) 26601M->26582M(26624M) 2621,898ms
[2026-06-05T08:21:06.655+0300][3571566][gc] GC(97) Concurrent Mark Cycle 2623,485ms
[2026-06-05T08:21:06.685+0300][3571566][gc] GC(98) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 26614M->26614M(26624M) 11,518ms
Проблема найдена. Вы ошиблись в настройках — IDM работает не на ZGC, а на G1GC.
Лог чётко показывает:
· Pause Young (Normal) → паузы G1
· Evacuation Failure → регионы кучи переполнены, G1 не успевает их очистить
· Pause Full (G1 Compaction Pause) → фатальные Full GC по 2–5 секунд
Что происходит
Куча 26 ГБ забита под завязку (26615M->26615M(26624M) — использовано 99.97%). G1 пытается копировать живые объекты во время молодых сборок, но свободных регионов нет — всё занято. Начинаются бесконечные Full GC, которые не могут ничего очистить (память падает с 26615 до 26593 МБ — копейки).
CPU простаивает, потому что приложение ждёт GC в блокировках.
Причина
G1 не справляется с объёмом живых данных. Ваше приложение держит в памяти 25+ ГБ живых объектов (кеши, сессии, невыгруженные данные). G1 для такого размера кучи даёт сбои — он не предназначен для гепа >20 ГБ с высоким процентом живущих объектов.
Немедленное решение
Переключитесь на ZGC (как и планировалось):
```bash
-XX:+UseZGC -Xmx24G -Xms24G \
-XX:ConcGCThreads=8 \
-XX:ZCollectionInterval=30 \
-XX:ZAllocationSpikeTolerance=5
```
ZGC умеет работать с кучей 24+ ГБ и не даёт длинных пауз.
Проверьте, что флаги применились:
```bash
jcmd $(pgrep -f idm) VM.flags | grep UseZGC
```
Если ZGC не поможет
Значит, приложение действительно требует >25 ГБ живых данных. Тогда:
1. Увеличить -Xmx до 32 ГБ (но осторожно, останется мало памяти ОС)
2. Или найти утечку через дамп кучи:
jmap -dump:live,format=b,file=heap.hprof <PID>
Приложите полную командную строку JVM — проверю, почему ZGC не включился.
Настройки сейчас вот такие:
-Djava.net.preferIPv4Stack=true \
-XX:+UseZGC \
-XX:+ZGenerational \
-XX:SoftMaxHeapSize=10g \
-XX:ZAllocationSpikeTolerance=8 \
-XX:ConcGCThreads=8 \
-XX:ParallelGCThreads=24 \
-XX:+ZProactive \
-XX:+AlwaysPreTouch \
-XX:+ZUncommit \
-XX:ZUncommitDelay=600 \
-XX:+UseNUMA \
-XX:MaxMetaspaceSize=512m \
-XX:MetaspaceSize=256m \
-XX:+PrintCommandLineFlags \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/opt/ankey/tmp \
-XX:NativeMemoryTracking=detail
-XX:+ExitOnOutOfMemoryError \
Сейчас покажу лог gc
Всё стало понятно. ZGC включён, но лог, который вы прислали ранее — от G1. Это значит, что процесс был перезапущен между сменой настроек. Сейчас ZGC работает, но падения продолжаются — и вот почему.
Ключевая проблема в ваших настройках
-XX:SoftMaxHeapSize=10g — это ловушка!
ZGC при этом флаге:
· Держит кучу физически на уровне Xmx (у вас не указан, значит по умолчанию RAM/2 ~ 23.5 ГБ)
· Но пытается урвать память обратно в ОС, как только живая куча опускается ниже 10 ГБ
· При аллокациях выше 10 ГБ начинаются частые циклы маркировки, которые не успевают за приложением → allocation stalls → OOM
Что происходит на самом деле
У вас Xmx по умолчанию ~23.5 ГБ (47 ГБ / 2).
ZGC видит: «живых данных >10 ГБ? Надо срожать кучу до 10 ГБ!»
Но приложение их не отдаёт → бесконечные циклы маркировки и сталы.
Немедленное решение
Замените SoftMaxHeapSize на явный Xmx:
```bash
-Xmx20G -Xms20G \
-XX:+UseZGC \
-XX:+ZGenerational \
-XX:ConcGCThreads=8 \
-XX:ParallelGCThreads=20 \
-XX:ZCollectionInterval=60 \
-XX:ZAllocationSpikeTolerance=8 \
-XX:+AlwaysPreTouch \
-XX:+UseNUMA \
-XX:MaxMetaspaceSize=512m
```
Уберите:
· -XX:SoftMaxHeapSize=10g (это для экспериментов с экономией памяти)
· -XX:+ZUncommit (медленно работает на больших кучах)
· -XX:+ZProactive (мешает при пиковых нагрузках)
Пришлите свежий лог ZGC
После перезапуска с правильными настройками выполните и покажите:
```bash
grep -E "Allocation stall|OutOfMemory|used=" /path/to/gc.log | tail -30
```
Увидим, ушли ли сталы.
А пока — ваши настройки гарантированно вызывают OOM через 5-10 минут при любой нагрузке выше 10 ГБ.
Отлично. Жду лог.
Пока IDM работает, можете параллельно собрать диагностику "до греха" (пригодятся, если падение повторится):
1. Текущее состояние памяти ZGC
```bash
jstat -gc $(pgrep -f idm) 1s 10
```
Обратите внимание на столбцы:
· ZHeap — занятость кучи
· ZOld* / ZYoung* (если включён ZGenerational)
2. Аллокационные сталы (живой мониторинг)
```bash
tail -f /path/to/gc.log | grep --line-buffered "Allocation stall"
```
Если появятся строки с задержками >100 мс — сигнал, что ZGC опять не успевает.
3. CPU-профиль (не простаивает ли?)
```bash
top -H -p $(pgrep -f idm)
```
При нормальной работе ZGC треды ZGC* должны потреблять 10–30% CPU. Если они близки к нулю — проблема в I/O или блокировках самого IDM.
Когда пришлёте лог, обращу внимание на:
· Частоту Concurrent Mark Cycle — должна быть не чаще раза в 5–10 секунд
· Allocation stall — их быть не должно
· High vs Used в фазе маркировки — разрыв не должен превышать 5–10%
Пока лог не пришёл: проверьте, что Xmx действительно применился:
```bash
jcmd $(pgrep -f idm) VM.flags | grep -E "Xmx|UseZGC"
```
Жду лог и jstat — диагноз будет готов через минуту после получения.
[2026-06-05T08:31:43.472+0300][3574699][gc] Using The Z Garbage Collector
[2026-06-05T08:32:20.662+0300][3574699][gc] GC(0) Major Collection (CodeCache GC Threshold)
[2026-06-05T08:32:21.414+0300][3574699][gc] GC(0) Major Collection (CodeCache GC Threshold) 1796M(7%)->408M(2%) 0,753s
[2026-06-05T08:32:28.138+0300][3574699][gc] GC(1) Major Collection (Warmup)
[2026-06-05T08:32:28.284+0300][3574699][gc] GC(1) Major Collection (Warmup) 2666M(10%)->272M(1%) 0,146s
[2026-06-05T08:33:28.290+0300][3574699][gc] GC(2) Major Collection (Timer)
[2026-06-05T08:33:28.621+0300][3574699][gc] GC(2) Major Collection (Timer) 2204M(8%)->234M(1%) 0,331s
[2026-06-05T08:34:28.626+0300][3574699][gc] GC(3) Major Collection (Timer)
[2026-06-05T08:34:28.949+0300][3574699][gc] GC(3) Major Collection (Timer) 238M(1%)->224M(1%) 0,323s
[2026-06-05T08:35:28.957+0300][3574699][gc] GC(4) Major Collection (Timer)
[2026-06-05T08:35:29.293+0300][3574699][gc] GC(4) Major Collection (Timer) 228M(1%)->214M(1%) 0,336s
[2026-06-05T08:36:17.483+0300][3574699][gc] GC(5) Major Collection (Warmup)
[2026-06-05T08:36:17.946+0300][3574699][gc] GC(5) Major Collection (Warmup) 5326M(20%)->600M(2%) 0,463s
[2026-06-05T08:37:01.463+0300][3574699][gc] GC(6) Major Collection (CodeCache GC Threshold)
[2026-06-05T08:37:02.790+0300][3574699][gc] GC(6) Major Collection (CodeCache GC Threshold) 4124M(15%)->860M(3%) 1,328s
[2026-06-05T08:37:20.403+0300][3574699][gc] GC(7) Major Collection (Warmup)
[2026-06-05T08:37:21.761+0300][3574699][gc] GC(7) Major Collection (Warmup) 7990M(30%)->5760M(22%) 1,359s
[2026-06-05T08:37:24.407+0300][3574699][gc] GC(8) Minor Collection (Allocation Rate)
[2026-06-05T08:37:29.190+0300][3574699][gc] GC(8) Minor Collection (Allocation Rate) 11526M(43%)->19138M(72%) 4,784s
[2026-06-05T08:37:29.191+0300][3574699][gc] GC(9) Major Collection (Allocation Rate)
[2026-06-05T08:37:33.318+0300][3574699][gc] Allocation Stall (parallel-7) 217,997ms
[2026-06-05T08:37:33.318+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 217,784ms
[2026-06-05T08:37:35.407+0300][3574699][gc] GC(10) Minor Collection (Allocation Stall)
[2026-06-05T08:37:38.499+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 4069,385ms
[2026-06-05T08:37:38.499+0300][3574699][gc] Allocation Stall (parallel-7) 4068,207ms
[2026-06-05T08:37:39.175+0300][3574699][gc] Allocation Stall (httpclient-dispatch-10) 4423,267ms
[2026-06-05T08:37:39.175+0300][3574699][gc] Allocation Stall (Reference Handler) 3583,488ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (Common-Cleaner) 3583,707ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-13) 3420,717ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-6) 4042,090ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-11) 4425,455ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-12) 3422,064ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-16) 3423,192ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-6) 4419,593ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-18) 4418,589ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-1-1) 4197,247ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 3559,034ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (fileinstall-./load) 3411,417ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-19) 3422,095ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-9) 3419,727ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-7) 3416,078ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 3368,375ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (qtp342201216-155-acceptor-0@5b8a41c0-0.0.0.0:8080@322f8660{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 3366,243ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (httpclient-dispatch-4) 4422,541ms
[2026-06-05T08:37:39.176+0300][3574699][gc] Allocation Stall (fileinstall-/opt/ankey/idm/conf) 3352,569ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (fileinstall-/opt/ankey/idm/ui/default) 3412,111ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 3261,013ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-5) 3259,504ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-8) 3259,357ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-20) 3260,364ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-17) 3257,965ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-15) 3259,418ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-2) 3045,497ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-14) 3261,124ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-1) 3134,306ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (httpclient-dispatch-3) 3256,545ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (parallel-1) 2969,529ms
[2026-06-05T08:37:39.177+0300][3574699][gc] Allocation Stall (parallel-3) 2919,624ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (parallel-5) 2884,223ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (parallel-4) 2899,558ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (parallel-6) 2867,923ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (fileinstall-/opt/ankey/idm/workflow) 2798,708ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (parallel-2) 2936,677ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (OpenICF Client ConnectionManager Scheduler 2) 177,000ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (parallel-7) 678,151ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (OpenICF Client ConnectionManager Scheduler 0) 177,007ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-4) 1,904ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (qtp342201216-231) 0,708ms
[2026-06-05T08:37:39.178+0300][3574699][gc] Allocation Stall (qtp342201216-156-acceptor-1@22bb0f3b-0.0.0.0:8080@322f8660{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 1,073ms
[2026-06-05T08:37:39.180+0300][3574699][gc] Relocation Stall (pool-7-thread-1) 1,871ms
[2026-06-05T08:37:39.180+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 675,258ms
[2026-06-05T08:37:39.184+0300][3574699][gc] Allocation Stall (parallel-7) 5,476ms
[2026-06-05T08:37:40.302+0300][3574699][gc] GC(10) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 4,894s
[2026-06-05T08:37:40.302+0300][3574699][gc] GC(11) Minor Collection (Allocation Stall)
[2026-06-05T08:37:42.072+0300][3574699][gc] Allocation Stall (parallel-7) 1917,484ms
[2026-06-05T08:37:42.072+0300][3574699][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 1394,336ms
[2026-06-05T08:37:42.072+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-5) 1885,605ms
[2026-06-05T08:37:42.074+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 1917,529ms
[2026-06-05T08:37:42.074+0300][3574699][gc] Allocation Stall (OpenICF Client ConnectionManager Scheduler 1) 1212,875ms
[2026-06-05T08:37:42.923+0300][3574699][gc] Allocation Stall (fileinstall-./load) 1746,373ms
[2026-06-05T08:37:42.923+0300][3574699][gc] Allocation Stall (OpenICF Client ConnectionManager Scheduler 3) 2062,028ms
[2026-06-05T08:37:42.923+0300][3574699][gc] Allocation Stall (httpclient-dispatch-10) 1745,337ms
[2026-06-05T08:37:42.923+0300][3574699][gc] Allocation Stall (httpclient-dispatch-9) 1745,332ms
[2026-06-05T08:37:42.923+0300][3574699][gc] Allocation Stall (fileinstall-/opt/ankey/idm/ui/default) 1746,154ms
[2026-06-05T08:37:42.923+0300][3574699][gc] Allocation Stall (httpclient-dispatch-7) 1745,355ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (fileinstall-/opt/ankey/idm/workflow) 1745,557ms
[2026-06-05T08:37:42.923+0300][3574699][gc] Allocation Stall (httpclient-dispatch-3) 1745,252ms
[2026-06-05T08:37:42.923+0300][3574699][gc] Allocation Stall (httpclient-dispatch-6) 1745,361ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (httpclient-dispatch-13) 1745,638ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (httpclient-dispatch-8) 1745,697ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (httpclient-dispatch-18) 1745,653ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (httpclient-dispatch-4) 1745,854ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (httpclient-dispatch-11) 1746,172ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (httpclient-dispatch-16) 1746,253ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (httpclient-dispatch-5) 1745,683ms
[2026-06-05T08:37:42.924+0300][3574699][gc] Allocation Stall (httpclient-dispatch-1) 1745,706ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (httpclient-dispatch-2) 1745,810ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (httpclient-dispatch-17) 1745,825ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (httpclient-dispatch-14) 1745,820ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (httpclient-dispatch-15) 1745,848ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (httpclient-dispatch-20) 1745,344ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (fileinstall-/opt/ankey/idm/conf) 1721,162ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (qtp342201216-155-acceptor-0@5b8a41c0-0.0.0.0:8080@322f8660{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 1719,686ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (parallel-1) 1717,228ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (parallel-3) 1667,239ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (parallel-4) 1647,093ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-1-1) 1645,979ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 1616,829ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (parallel-5) 1631,984ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (parallel-6) 1615,494ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 1609,091ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (pool-37-thread-1) 1603,003ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-4) 853,599ms
[2026-06-05T08:37:42.926+0300][3574699][gc] Allocation Stall (httpclient-dispatch-12) 1747,341ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (httpclient-dispatch-19) 1747,125ms
[2026-06-05T08:37:42.925+0300][3574699][gc] Allocation Stall (parallel-2) 1684,109ms
[2026-06-05T08:37:42.928+0300][3574699][gc] Allocation Stall (parallel-7) 855,247ms
[2026-06-05T08:37:42.928+0300][3574699][gc] Allocation Stall (OpenICF Client Grizzly selector thread(1) SelectorRunner) 854,089ms
[2026-06-05T08:37:42.929+0300][3574699][gc] Allocation Stall (OpenICF Client Grizzly selector thread(2) SelectorRunner) 854,134ms
[2026-06-05T08:37:42.929+0300][3574699][gc] Allocation Stall (qtp342201216-156-acceptor-1@22bb0f3b-0.0.0.0:8080@322f8660{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 3,434ms
[2026-06-05T08:37:42.930+0300][3574699][gc] Relocation Stall (pool-7-thread-1) 1,023ms
[2026-06-05T08:37:42.930+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 855,412ms
[2026-06-05T08:37:43.908+0300][3574699][gc] GC(11) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 3,606s
[2026-06-05T08:37:43.908+0300][3574699][gc] GC(12) Minor Collection (Allocation Stall)
[2026-06-05T08:37:44.344+0300][3574699][gc] Allocation Stall (parallel-7) 1139,660ms
[2026-06-05T08:37:44.495+0300][3574699][gc] Allocation Stall (Notification Thread) 586,901ms
[2026-06-05T08:37:44.495+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-6) 655,686ms
[2026-06-05T08:37:44.495+0300][3574699][gc] Allocation Stall (httpclient-dispatch-10) 570,834ms
[2026-06-05T08:37:44.495+0300][3574699][gc] Allocation Stall (httpclient-dispatch-9) 571,037ms
[2026-06-05T08:37:44.495+0300][3574699][gc] Allocation Stall (httpclient-dispatch-7) 570,771ms
[2026-06-05T08:37:44.495+0300][3574699][gc] Allocation Stall (httpclient-dispatch-6) 570,862ms
[2026-06-05T08:37:44.495+0300][3574699][gc] Allocation Stall (httpclient-dispatch-3) 570,828ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-18) 570,681ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-13) 570,740ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-8) 571,097ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-4) 570,801ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-5) 570,382ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-17) 570,482ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-11) 571,038ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-20) 570,490ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-16) 570,440ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Relocation Stall (pool-7-thread-1) 1,011ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 1289,387ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-2) 570,428ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-14) 570,458ms
[2026-06-05T08:37:44.496+0300][3574699][gc] Allocation Stall (httpclient-dispatch-15) 570,433ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (httpclient-dispatch-12) 570,197ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (httpclient-dispatch-19) 570,489ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-1) 563,405ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-5) 562,262ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 563,615ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 559,923ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-4) 563,177ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 560,918ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-2) 556,246ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 557,706ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 554,621ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 553,421ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (httpclient-dispatch-1) 520,465ms
[2026-06-05T08:37:44.498+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 489,906ms
[2026-06-05T08:37:44.498+0300][3574699][gc] Allocation Stall (commons-pool-evictor) 250,505ms
[2026-06-05T08:37:44.498+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 480,840ms
[2026-06-05T08:37:44.498+0300][3574699][gc] Allocation Stall (qtp342201216-155-acceptor-0@5b8a41c0-0.0.0.0:8080@322f8660{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 217,401ms
[2026-06-05T08:37:44.497+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-3) 556,436ms
[2026-06-05T08:37:44.498+0300][3574699][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 147,190ms
[2026-06-05T08:37:44.498+0300][3574699][gc] Allocation Stall (AsyncHttpClient-timer-1-1) 519,647ms
[2026-06-05T08:37:44.499+0300][3574699][gc] Allocation Stall (parallel-7) 153,946ms
[2026-06-05T08:37:44.499+0300][3574699][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 2,053ms
[2026-06-05T08:37:44.501+0300][3574699][gc] Relocation Stall (reactive-kafka-ankey_default-5) 4,087ms
[2026-06-05T08:37:44.502+0300][3574699][gc] Relocation Stall (reactive-kafka-ankey_default-2) 4,509ms
[2026-06-05T08:37:44.753+0300][3574699][gc] GC(12) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,846s
[2026-06-05T08:37:44.753+0300][3574699][gc] GC(13) Minor Collection (Allocation Stall)
[2026-06-05T08:37:44.882+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 326,335ms
[2026-06-05T08:37:44.882+0300][3574699][gc] Allocation Stall (Notification Thread) 128,245ms
[2026-06-05T08:37:44.882+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 82,717ms
[2026-06-05T08:37:44.882+0300][3574699][gc] Allocation Stall (parallel-7) 325,612ms
[2026-06-05T08:37:44.932+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-4) 132,543ms
[2026-06-05T08:37:44.932+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-3) 131,924ms
[2026-06-05T08:37:44.932+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 132,180ms
[2026-06-05T08:37:44.932+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-6) 132,107ms
[2026-06-05T08:37:44.932+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 132,593ms
[2026-06-05T08:37:44.932+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-1) 131,714ms
[2026-06-05T08:37:44.933+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 131,993ms
[2026-06-05T08:37:44.933+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 130,031ms
[2026-06-05T08:37:44.933+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-2) 130,392ms
[2026-06-05T08:37:44.933+0300][3574699][gc] Relocation Stall (qtp342201216-156-acceptor-1@22bb0f3b-0.0.0.0:8080@322f8660{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 0,520ms
[2026-06-05T08:37:44.933+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 129,756ms
[2026-06-05T08:37:44.933+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-5) 129,881ms
[2026-06-05T08:37:44.935+0300][3574699][gc] Allocation Stall (parallel-7) 52,297ms
[2026-06-05T08:37:44.935+0300][3574699][gc] Allocation Stall (oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser) 52,210ms
[2026-06-05T08:37:44.936+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 49,230ms
[2026-06-05T08:37:44.937+0300][3574699][gc] Allocation Stall (fileinstall-./load) 13,669ms
[2026-06-05T08:37:44.938+0300][3574699][gc] Allocation Stall (qtp342201216-156-acceptor-1@22bb0f3b-0.0.0.0:8080@322f8660{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 4,546ms
[2026-06-05T08:37:45.015+0300][3574699][gc] GC(13) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,262s
[2026-06-05T08:37:45.015+0300][3574699][gc] GC(14) Minor Collection (Allocation Stall)
[2026-06-05T08:37:45.066+0300][3574699][gc] Allocation Stall (parallel-7) 126,637ms
[2026-06-05T08:37:45.083+0300][3574699][gc] Allocation Stall (qtp342201216-145) 87,399ms
[2026-06-05T08:37:45.083+0300][3574699][gc] Allocation Stall (qtp342201216-231) 120,423ms
[2026-06-05T08:37:45.083+0300][3574699][gc] Allocation Stall (qtp342201216-405) 142,593ms
[2026-06-05T08:37:45.083+0300][3574699][gc] Allocation Stall (C1 CompilerThread0) 80,615ms
[2026-06-05T08:37:45.084+0300][3574699][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 85,331ms
[2026-06-05T08:37:45.084+0300][3574699][gc] Relocation Stall (qtp342201216-145) 0,451ms
[2026-06-05T08:37:45.084+0300][3574699][gc] Allocation Stall (Notification Thread) 68,411ms
[2026-06-05T08:37:45.084+0300][3574699][gc] Relocation Stall (pool-7-thread-1) 0,610ms
[2026-06-05T08:37:45.084+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 144,606ms
[2026-06-05T08:37:45.084+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 51,060ms
[2026-06-05T08:37:45.084+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 51,189ms
[2026-06-05T08:37:45.114+0300][3574699][gc] GC(14) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,099s
[2026-06-05T08:37:45.114+0300][3574699][gc] GC(15) Minor Collection (Allocation Stall)
[2026-06-05T08:37:45.147+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 114,219ms
[2026-06-05T08:37:45.148+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 114,800ms
[2026-06-05T08:37:45.148+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-4) 114,835ms
[2026-06-05T08:37:45.148+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 115,010ms
[2026-06-05T08:37:45.150+0300][3574699][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 1,520ms
[2026-06-05T08:37:45.150+0300][3574699][gc] Relocation Stall (reactive-kafka-ankey_default-6) 1,571ms
[2026-06-05T08:37:45.150+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-2) 116,110ms
[2026-06-05T08:37:45.150+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-6) 116,649ms
[2026-06-05T08:37:45.150+0300][3574699][gc] GC(15) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,036s
[2026-06-05T08:37:45.175+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-5) 141,342ms
[2026-06-05T08:37:45.175+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-3) 141,450ms
[2026-06-05T08:37:45.175+0300][3574699][gc] Allocation Stall (reactive-kafka-ankey_default-1) 141,616ms
[2026-06-05T08:37:45.176+0300][3574699][gc] GC(16) Minor Collection (Allocation Stall)
[2026-06-05T08:37:45.177+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 142,523ms
[2026-06-05T08:37:45.215+0300][3574699][gc] Allocation Stall (parallel-7) 148,016ms
[2026-06-05T08:37:45.216+0300][3574699][gc] GC(16) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,040s
[2026-06-05T08:37:45.217+0300][3574699][gc] GC(9) Major Collection (Allocation Rate) 19138M(72%)->26624M(100%) 16,026s
[2026-06-05T08:37:45.217+0300][3574699][gc] GC(17) Major Collection (Allocation Stall)
[2026-06-05T08:37:45.246+0300][3574699][gc] Allocation Stall (qtp342201216-406) 162,722ms
[2026-06-05T08:37:45.246+0300][3574699][gc] Allocation Stall (Connector-Scheduler-322f8660-1) 162,560ms
[2026-06-05T08:37:45.248+0300][3574699][gc] Allocation Stall (qtp342201216-152) 163,603ms
[2026-06-05T08:37:45.277+0300][3574699][gc] Allocation Stall (ForkJoinPool.commonPool-worker-7) 191,218ms
[2026-06-05T08:37:45.277+0300][3574699][gc] Allocation Stall (pool-7-thread-1) 188,924ms
[2026-06-05T08:37:45.278+0300][3574699][gc] GC(18) Minor Collection (Allocation Stall)
[2026-06-05T08:37:45.314+0300][3574699][gc] GC(18) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,036s
[2026-06-05T08:37:45.497+0300][3574699][gc] GC(19) Minor Collection (Allocation Stall)
[2026-06-05T08:37:45.530+0300][3574699][gc] GC(19) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,033s
[2026-06-05T08:37:45.579+0300][3574699][gc] GC(20) Minor Collection (Allocation Stall)
[2026-06-05T08:37:45.611+0300][3574699][gc] GC(20) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,032s
[2026-06-05T08:37:46.208+0300][3574699][gc] GC(21) Minor Collection (Allocation Stall)
[2026-06-05T08:37:46.241+0300][3574699][gc] GC(21) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,033s
[2026-06-05T08:37:46.241+0300][3574699][gc] GC(22) Minor Collection (Allocation Stall)
[2026-06-05T08:37:46.273+0300][3574699][gc] GC(22) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,032s
[2026-06-05T08:37:46.278+0300][3574699][gc] GC(23) Minor Collection (Allocation Stall)
[2026-06-05T08:37:46.310+0300][3574699][gc] Allocation Stall (Notification Thread) 1196,030ms
[2026-06-05T08:37:46.310+0300][3574699][gc] GC(23) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,032s
[2026-06-05T08:37:46.311+0300][3574699][gc] GC(24) Minor Collection (Allocation Stall)
[2026-06-05T08:37:46.342+0300][3574699][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 1193,801ms
[2026-06-05T08:37:46.345+0300][3574699][gc] GC(24) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,034s
[2026-06-05T08:37:46.345+0300][3574699][gc] GC(25) Minor Collection (Allocation Stall)
[2026-06-05T08:37:46.378+0300][3574699][gc] GC(25) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,033s
[2026-06-05T08:37:46.445+0300][3574699][gc] GC(26) Minor Collection (Allocation Stall)
[2026-06-05T08:37:46.480+0300][3574699][gc] GC(26) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,036s
[2026-06-05T08:37:46.924+0300][3574699][gc] GC(27) Minor Collection (Allocation Stall)
[2026-06-05T08:37:46.957+0300][3574699][gc] GC(27) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,033s
[2026-06-05T08:37:46.966+0300][3574699][gc] GC(28) Minor Collection (Allocation Stall)
[2026-06-05T08:37:46.999+0300][3574699][gc] GC(28) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,033s
[2026-06-05T08:37:47.765+0300][3574699][gc] GC(29) Minor Collection (Allocation Stall)
[2026-06-05T08:37:47.800+0300][3574699][gc] GC(29) Minor Collection (Allocation Stall) 26624M(100%)->26624M(100%) 0,034s
Диагноз абсолютно ясен.
Прямое показание лога
Строка 26624M(100%)->26624M(100%) повторяется раз за разом. Это значит:
Куча заполнена на 100% и ZGC не может освободить ни байта.
ZGC в генерационном режиме пытается делать Minor Collection (молодую сборку), но она не находит мёртвых объектов — всё, что в куче, живое. Через 4 секунды после старта приложение уже упёрлось в потолок.
Что это значит на практике
Вашему IDM нужно больше 26 ГБ живых данных постоянно. Это не всплеск, не утечка (пока) — это рабочий объём.
Причина почти наверняка:
· Кэши IDM (сессии, результаты синхронизации, workflow)
· Буферы Kafka (reactive-kafka держит огромные batch'и)
· OpenSearch client (буферизует ответы)
Немедленное решение (единственное работающее)
Увеличьте Xmx до 32–35 ГБ, если позволяет стенд:
```bash
-Xmx32G -Xms32G \
-XX:+UseZGC \
-XX:+ZGenerational \
-XX:ConcGCThreads=12 \
-XX:ParallelGCThreads=20 \
-XX:ZCollectionInterval=60 \
-XX:+AlwaysPreTouch \
-XX:+UseNUMA
```
Проверьте свободную память до старта:
```bash
free -h
```
На 47 ГБ стенда 32 ГБ для IDM — предел. Если останется <10 ГБ на Kafka+OS+OpenSearch, нужно разносить сервисы.
Если нельзя увеличить кучу — нужно резать рабочий объём
1. Уменьшить кэши IDM (найти в конфиге параметры типа cache.max.entries, session.timeout)
2. Ограничить Kafka consumer (max.poll.records=100 вместо 500)
3. Отключить ненужные модули IDM
Что смотреть дальше
После увеличения кучи (или без него) сделайте дамп:
```bash
jcmd $(pgrep -f idm) GC.heap_info
```
Покажите конфигурацию IDM (особенно кэши и Kafka-настройки) — подскажу, что резать.
Вывод: проблема не в GC, проблема в том, что приложению нужно 30+ ГБ памяти. Увеличивайте кучу.
[2026-06-05T08:56:28.793+0300][3579309][gc] Using The Z Garbage Collector
[2026-06-05T08:56:40.874+0300][3579309][gc] GC(0) Major Collection (CodeCache GC Threshold)
[2026-06-05T08:56:42.222+0300][3579309][gc] GC(0) Major Collection (CodeCache GC Threshold) 1828M(6%)->442M(1%) 1,348s
[2026-06-05T08:56:48.751+0300][3579309][gc] GC(1) Major Collection (Warmup)
[2026-06-05T08:56:48.910+0300][3579309][gc] GC(1) Major Collection (Warmup) 3072M(10%)->288M(1%) 0,159s
[2026-06-05T08:57:48.912+0300][3579309][gc] GC(2) Major Collection (Timer)
[2026-06-05T08:57:49.296+0300][3579309][gc] GC(2) Major Collection (Timer) 1848M(6%)->240M(1%) 0,385s
[2026-06-05T08:58:49.303+0300][3579309][gc] GC(3) Major Collection (Timer)
[2026-06-05T08:58:49.679+0300][3579309][gc] GC(3) Major Collection (Timer) 244M(1%)->228M(1%) 0,375s
[2026-06-05T08:59:49.680+0300][3579309][gc] GC(4) Major Collection (Timer)
[2026-06-05T08:59:50.028+0300][3579309][gc] GC(4) Major Collection (Timer) 232M(1%)->230M(1%) 0,348s
[2026-06-05T09:00:40.766+0300][3579309][gc] GC(5) Major Collection (CodeCache GC Threshold)
[2026-06-05T09:00:41.561+0300][3579309][gc] GC(5) Major Collection (CodeCache GC Threshold) 5594M(18%)->386M(1%) 0,795s
[2026-06-05T09:00:55.953+0300][3579309][gc] GC(6) Major Collection (CodeCache GC Threshold)
[2026-06-05T09:00:57.325+0300][3579309][gc] GC(6) Major Collection (CodeCache GC Threshold) 4152M(14%)->812M(3%) 1,373s
[2026-06-05T09:01:12.253+0300][3579309][gc] GC(7) Major Collection (Warmup)
[2026-06-05T09:01:13.310+0300][3579309][gc] GC(7) Major Collection (Warmup) 6144M(20%)->3582M(12%) 1,057s
[2026-06-05T09:01:20.555+0300][3579309][gc] GC(8) Major Collection (Warmup)
[2026-06-05T09:01:25.005+0300][3579309][gc] GC(9) Minor Collection (Allocation Rate)
[2026-06-05T09:01:29.118+0300][3579309][gc] GC(9) Minor Collection (Allocation Rate) 7778M(25%)->8862M(29%) 4,113s
[2026-06-05T09:01:29.118+0300][3579309][gc] GC(10) Minor Collection (Allocation Rate)
[2026-06-05T09:01:33.728+0300][3579309][gc] GC(10) Minor Collection (Allocation Rate) 8862M(29%)->9714M(32%) 4,609s
[2026-06-05T09:01:33.746+0300][3579309][gc] GC(11) Minor Collection (Allocation Rate)
[2026-06-05T09:01:36.239+0300][3579309][gc] GC(11) Minor Collection (Allocation Rate) 9714M(32%)->8988M(29%) 2,494s
[2026-06-05T09:01:36.239+0300][3579309][gc] GC(8) Major Collection (Warmup) 9218M(30%)->8988M(29%) 15,684s
[2026-06-05T09:01:36.249+0300][3579309][gc] GC(12) Major Collection (Allocation Rate)
[2026-06-05T09:01:40.754+0300][3579309][gc] GC(13) Minor Collection (Allocation Rate)
[2026-06-05T09:01:41.575+0300][3579309][gc] GC(13) Minor Collection (Allocation Rate) 11164M(36%)->9228M(30%) 0,821s
[2026-06-05T09:01:41.575+0300][3579309][gc] GC(14) Minor Collection (Allocation Rate)
[2026-06-05T09:01:42.653+0300][3579309][gc] GC(14) Minor Collection (Allocation Rate) 9228M(30%)->9626M(31%) 1,078s
[2026-06-05T09:01:43.244+0300][3579309][gc] GC(15) Minor Collection (Allocation Rate)
[2026-06-05T09:01:43.557+0300][3579309][gc] GC(15) Minor Collection (Allocation Rate) 10088M(33%)->9346M(30%) 0,312s
[2026-06-05T09:01:44.494+0300][3579309][gc] GC(16) Minor Collection (Allocation Rate)
[2026-06-05T09:01:45.125+0300][3579309][gc] GC(16) Minor Collection (Allocation Rate) 10084M(33%)->9902M(32%) 0,631s
[2026-06-05T09:01:45.556+0300][3579309][gc] GC(17) Minor Collection (Allocation Rate)
[2026-06-05T09:01:46.237+0300][3579309][gc] GC(17) Minor Collection (Allocation Rate) 10268M(33%)->10018M(33%) 0,680s
[2026-06-05T09:01:57.524+0300][3579309][gc] GC(12) Major Collection (Allocation Rate) 8996M(29%)->15020M(49%) 21,274s
[2026-06-05T09:02:01.589+0300][3579309][gc] GC(18) Major Collection (Allocation Rate)
[2026-06-05T09:02:03.667+0300][3579309][gc] GC(19) Minor Collection (Allocation Rate)
[2026-06-05T09:02:05.528+0300][3579309][gc] GC(19) Minor Collection (Allocation Rate) 13912M(45%)->15092M(49%) 1,862s
[2026-06-05T09:02:05.529+0300][3579309][gc] GC(20) Minor Collection (Allocation Rate)
[2026-06-05T09:02:06.242+0300][3579309][gc] GC(20) Minor Collection (Allocation Rate) 15092M(49%)->14104M(46%) 0,713s
[2026-06-05T09:02:06.242+0300][3579309][gc] GC(21) Minor Collection (Allocation Rate)
[2026-06-05T09:02:07.666+0300][3579309][gc] GC(21) Minor Collection (Allocation Rate) 14104M(46%)->17080M(56%) 1,424s
[2026-06-05T09:02:07.666+0300][3579309][gc] GC(22) Minor Collection (Allocation Rate)
[2026-06-05T09:02:09.380+0300][3579309][gc] GC(22) Minor Collection (Allocation Rate) 17080M(56%)->18294M(60%) 1,714s
[2026-06-05T09:02:09.380+0300][3579309][gc] GC(23) Minor Collection (Allocation Rate)
[2026-06-05T09:02:11.017+0300][3579309][gc] GC(23) Minor Collection (Allocation Rate) 18296M(60%)->22412M(73%) 1,637s
[2026-06-05T09:02:11.017+0300][3579309][gc] GC(24) Minor Collection (Allocation Rate)
[2026-06-05T09:02:12.503+0300][3579309][gc] GC(24) Minor Collection (Allocation Rate) 22412M(73%)->24942M(81%) 1,486s
[2026-06-05T09:02:12.504+0300][3579309][gc] GC(25) Minor Collection (Allocation Rate)
[2026-06-05T09:02:14.952+0300][3579309][gc] GC(25) Minor Collection (Allocation Rate) 24942M(81%)->28514M(93%) 2,448s
[2026-06-05T09:02:14.952+0300][3579309][gc] GC(26) Minor Collection (Allocation Rate)
[2026-06-05T09:02:16.426+0300][3579309][gc] Allocation Stall (parallel-8) 662,480ms
[2026-06-05T09:02:16.426+0300][3579309][gc] Allocation Stall (httpclient-dispatch-1) 516,543ms
[2026-06-05T09:02:16.426+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 306,148ms
[2026-06-05T09:02:16.430+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 640,795ms
[2026-06-05T09:02:16.793+0300][3579309][gc] GC(26) Minor Collection (Allocation Rate) 28514M(93%)->29304M(95%) 1,841s
[2026-06-05T09:02:16.793+0300][3579309][gc] GC(27) Minor Collection (Allocation Rate)
[2026-06-05T09:02:17.505+0300][3579309][gc] Allocation Stall (parallel-8) 255,749ms
[2026-06-05T09:02:17.505+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 254,893ms
[2026-06-05T09:02:17.505+0300][3579309][gc] Allocation Stall (httpclient-dispatch-1) 77,588ms
[2026-06-05T09:02:17.505+0300][3579309][gc] Allocation Stall (Reference Handler) 0,882ms
[2026-06-05T09:02:17.506+0300][3579309][gc] Allocation Stall (Common-Cleaner) 1,543ms
[2026-06-05T09:02:18.543+0300][3579309][gc] GC(27) Minor Collection (Allocation Rate) 29304M(95%)->30510M(99%) 1,750s
[2026-06-05T09:02:18.580+0300][3579309][gc] GC(28) Minor Collection (Allocation Rate)
[2026-06-05T09:02:18.633+0300][3579309][gc] Allocation Stall (parallel-8) 4,509ms
[2026-06-05T09:02:19.440+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 811,916ms
[2026-06-05T09:02:19.441+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 811,360ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-4) 0,110ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 803,957ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/workflow) 782,233ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-5) 777,573ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/ui/default) 782,350ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (parallel-8) 808,180ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-5) 0,096ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-6) 0,167ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-12) 776,005ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-4) 777,911ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-17) 773,777ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-3) 775,532ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-13) 775,648ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-2) 777,680ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 812,992ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (httpclient-dispatch-7) 774,579ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 0,487ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 813,067ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-19) 775,376ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 2,984ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 519,176ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 813,294ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-15) 773,733ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 803,731ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-16) 778,317ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-11) 777,628ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (httpclient-dispatch-6) 775,579ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (qtp881248216-156-acceptor-0@4a87020a-0.0.0.0:8080@2fc102ab{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 763,505ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (httpclient-dispatch-10) 775,643ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 811,516ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 810,670ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (httpclient-dispatch-8) 775,571ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-20) 775,195ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-1-1) 452,801ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 813,310ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (httpclient-dispatch-18) 774,140ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-9) 775,234ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/conf) 345,951ms
[2026-06-05T09:02:19.443+0300][3579309][gc] Allocation Stall (httpclient-dispatch-14) 775,387ms
[2026-06-05T09:02:19.444+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 504,933ms
[2026-06-05T09:02:19.455+0300][3579309][gc] GC(28) Minor Collection (Allocation Rate) 30586M(100%)->28574M(93%) 0,875s
[2026-06-05T09:02:19.455+0300][3579309][gc] GC(18) Major Collection (Allocation Rate) 18422M(60%)->28574M(93%) 17,866s
[2026-06-05T09:02:19.465+0300][3579309][gc] GC(29) Major Collection (Allocation Rate)
[2026-06-05T09:02:21.326+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 272,816ms
[2026-06-05T09:02:21.326+0300][3579309][gc] Allocation Stall (parallel-8) 272,133ms
[2026-06-05T09:02:21.665+0300][3579309][gc] Allocation Stall (parallel-8) 334,042ms
[2026-06-05T09:02:21.665+0300][3579309][gc] Allocation Stall (ForkJoinPool.commonPool-worker-6) 196,155ms
[2026-06-05T09:02:21.665+0300][3579309][gc] Allocation Stall (qtp881248216-154) 214,281ms
[2026-06-05T09:02:21.665+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 332,877ms
[2026-06-05T09:02:21.711+0300][3579309][gc] Allocation Stall (parallel-8) 4,212ms
[2026-06-05T09:02:21.711+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 4,050ms
[2026-06-05T09:02:21.737+0300][3579309][gc] Allocation Stall (parallel-8) 6,687ms
[2026-06-05T09:02:21.737+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 6,072ms
[2026-06-05T09:02:22.280+0300][3579309][gc] GC(30) Minor Collection (Allocation Stall)
[2026-06-05T09:02:22.847+0300][3579309][gc] Allocation Stall (parallel-8) 1090,747ms
[2026-06-05T09:02:22.847+0300][3579309][gc] Allocation Stall (commons-pool-evictor) 583,816ms
[2026-06-05T09:02:22.847+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 546,295ms
[2026-06-05T09:02:22.847+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 1089,440ms
[2026-06-05T09:02:22.848+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 546,786ms
[2026-06-05T09:02:22.848+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 546,828ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 547,356ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 547,425ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 547,495ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-6) 156,797ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-11) 157,045ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-10) 155,907ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 157,237ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 0,433ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (fileinstall-./load) 156,878ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-4) 0,115ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 547,293ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-1) 297,149ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-15) 154,294ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-1) 0,381ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-12) 154,710ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-7) 156,016ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-5) 0,247ms
[2026-06-05T09:02:22.850+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 547,656ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 0,174ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 547,173ms
[2026-06-05T09:02:22.850+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 24,503ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-3) 0,334ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-8) 156,115ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-20) 156,135ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-19) 154,356ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-3) 155,287ms
[2026-06-05T09:02:22.850+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 10,699ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-9) 154,285ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-5) 150,937ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-14) 156,402ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 0,397ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-17) 154,367ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-4) 156,422ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-16) 154,129ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-18) 154,366ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (parallel-7) 17,582ms
[2026-06-05T09:02:22.850+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 547,822ms
[2026-06-05T09:02:22.850+0300][3579309][gc] Allocation Stall (parallel-8) 2,008ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-13) 156,924ms
[2026-06-05T09:02:22.850+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 548,526ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-6) 0,427ms
[2026-06-05T09:02:22.849+0300][3579309][gc] Allocation Stall (httpclient-dispatch-2) 150,875ms
[2026-06-05T09:02:22.850+0300][3579309][gc] Relocation Stall (commons-pool-evictor) 1,151ms
[2026-06-05T09:02:22.851+0300][3579309][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 2,362ms
[2026-06-05T09:02:22.852+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 550,711ms
[2026-06-05T09:02:22.875+0300][3579309][gc] GC(30) Minor Collection (Allocation Stall) 30720M(100%)->29726M(97%) 0,595s
[2026-06-05T09:02:22.926+0300][3579309][gc] GC(31) Minor Collection (Allocation Rate)
[2026-06-05T09:02:23.351+0300][3579309][gc] Allocation Stall (parallel-8) 154,932ms
[2026-06-05T09:02:23.351+0300][3579309][gc] Allocation Stall (parallel-1) 124,016ms
[2026-06-05T09:02:23.351+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 155,585ms
[2026-06-05T09:02:23.351+0300][3579309][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 1,598ms
[2026-06-05T09:02:23.434+0300][3579309][gc] Allocation Stall (parallel-8) 78,215ms
[2026-06-05T09:02:23.434+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 78,506ms
[2026-06-05T09:02:23.605+0300][3579309][gc] GC(31) Minor Collection (Allocation Rate) 29868M(97%)->30720M(100%) 0,678s
[2026-06-05T09:02:23.605+0300][3579309][gc] GC(32) Minor Collection (Allocation Stall)
[2026-06-05T09:02:23.767+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 290,155ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 114,395ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser) 163,347ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 0,076ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (qtp881248216-157-acceptor-1@1274689e-0.0.0.0:8080@2fc102ab{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 100,313ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 114,018ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 112,424ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 114,270ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 114,108ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 0,541ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 114,317ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 112,597ms
[2026-06-05T09:02:23.770+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 112,471ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (parallel-8) 291,125ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 112,998ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 114,414ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-4) 0,214ms
[2026-06-05T09:02:23.769+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 114,555ms
[2026-06-05T09:02:23.770+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 115,017ms
[2026-06-05T09:02:23.773+0300][3579309][gc] GC(32) Minor Collection (Allocation Stall) 30720M(100%)->30342M(99%) 0,168s
[2026-06-05T09:02:23.909+0300][3579309][gc] GC(33) Minor Collection (Allocation Stall)
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 506,586ms
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 506,545ms
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 506,048ms
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 537,871ms
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 569,258ms
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 552,383ms
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (parallel-8) 559,648ms
[2026-06-05T09:02:24.479+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 506,735ms
[2026-06-05T09:02:24.479+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-1-1) 487,502ms
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 507,467ms
[2026-06-05T09:02:24.478+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 506,660ms
[2026-06-05T09:02:24.479+0300][3579309][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 127,495ms
[2026-06-05T09:02:24.480+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 1,571ms
[2026-06-05T09:02:24.481+0300][3579309][gc] Allocation Stall (parallel-8) 2,557ms
[2026-06-05T09:02:24.482+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 2,544ms
[2026-06-05T09:02:24.486+0300][3579309][gc] GC(33) Minor Collection (Allocation Stall) 30720M(100%)->30542M(99%) 0,577s
[2026-06-05T09:02:24.549+0300][3579309][gc] GC(34) Minor Collection (Allocation Stall)
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 837,250ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 837,651ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 836,550ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 837,763ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-10) 564,923ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Relocation Stall (pool-13-thread-1) 0,280ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-15) 565,192ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 838,171ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (parallel-5) 206,678ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 832,856ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (fileinstall-./load) 566,520ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-20) 564,861ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-12) 565,330ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-6) 564,915ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Relocation Stall (qtp881248216-157-acceptor-1@1274689e-0.0.0.0:8080@2fc102ab{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 0,205ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 0,191ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (qtp881248216-157-acceptor-1@1274689e-0.0.0.0:8080@2fc102ab{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 612,575ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-19) 563,969ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-3) 564,822ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-8) 564,932ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 836,787ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-14) 563,993ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-7) 565,419ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-1) 564,834ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-1) 0,340ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-3) 0,340ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 836,241ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-5) 563,820ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-4) 563,321ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-6) 0,031ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-9) 563,915ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 867,072ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 837,087ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-5) 0,094ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (httpclient-dispatch-13) 562,514ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (httpclient-dispatch-16) 563,020ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (httpclient-dispatch-2) 562,243ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (httpclient-dispatch-18) 562,951ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 206,784ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (httpclient-dispatch-17) 563,156ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (parallel-3) 205,765ms
[2026-06-05T09:02:25.417+0300][3579309][gc] Allocation Stall (parallel-4) 206,511ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 837,333ms
[2026-06-05T09:02:25.416+0300][3579309][gc] Allocation Stall (httpclient-dispatch-11) 565,355ms
[2026-06-05T09:02:25.419+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 839,463ms
[2026-06-05T09:02:25.419+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 838,217ms
[2026-06-05T09:02:25.419+0300][3579309][gc] Relocation Stall (parallel-8) 3,570ms
[2026-06-05T09:02:25.420+0300][3579309][gc] Allocation Stall (parallel-8) 869,905ms
[2026-06-05T09:02:25.421+0300][3579309][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 4,884ms
[2026-06-05T09:02:25.430+0300][3579309][gc] GC(34) Minor Collection (Allocation Stall) 30720M(100%)->30694M(100%) 0,881s
[2026-06-05T09:02:25.439+0300][3579309][gc] GC(35) Minor Collection (Allocation Stall)
[2026-06-05T09:02:25.678+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 238,270ms
[2026-06-05T09:02:25.678+0300][3579309][gc] Allocation Stall (qtp881248216-150) 237,485ms
[2026-06-05T09:02:25.678+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/workflow) 234,028ms
[2026-06-05T09:02:25.678+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/ui/default) 233,834ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 249,909ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 248,424ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 245,429ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 245,093ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Relocation Stall (AsyncHttpClient-timer-4-1) 0,159ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 249,606ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 245,173ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 245,289ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (C1 CompilerThread2) 228,472ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-3) 0,818ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 245,177ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 89,178ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Relocation Stall (AsyncHttpClient-timer-1-1) 0,476ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 228,891ms
[2026-06-05T09:02:25.770+0300][3579309][gc] GC(29) O: Relocation Stall (ZWorkerOld#7) 1,165ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 245,941ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 248,067ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 248,705ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/conf) 256,913ms
[2026-06-05T09:02:25.770+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-1-1) 278,899ms
[2026-06-05T09:02:25.768+0300][3579309][gc] Relocation Stall (AsyncHttpClient-timer-6-1) 0,115ms
[2026-06-05T09:02:25.770+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 1,989ms
[2026-06-05T09:02:25.769+0300][3579309][gc] Relocation Stall (qtp881248216-154) 0,127ms
[2026-06-05T09:02:25.771+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 245,203ms
[2026-06-05T09:02:25.771+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 253,363ms
[2026-06-05T09:02:25.771+0300][3579309][gc] Allocation Stall (qtp881248216-154) 91,507ms
[2026-06-05T09:02:25.772+0300][3579309][gc] GC(29) O: Relocation Stall (ZWorkerOld#0) 3,632ms
[2026-06-05T09:02:25.775+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-4) 6,799ms
[2026-06-05T09:02:25.777+0300][3579309][gc] Relocation Stall (parallel-8) 8,985ms
[2026-06-05T09:02:25.777+0300][3579309][gc] Allocation Stall (parallel-8) 322,868ms
[2026-06-05T09:02:25.823+0300][3579309][gc] Allocation Stall (parallel-8) 44,902ms
[2026-06-05T09:02:25.939+0300][3579309][gc] GC(35) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,499s
[2026-06-05T09:02:25.939+0300][3579309][gc] GC(36) Minor Collection (Allocation Stall)
[2026-06-05T09:02:26.255+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 456,789ms
[2026-06-05T09:02:26.255+0300][3579309][gc] Allocation Stall (parallel-8) 430,629ms
[2026-06-05T09:02:26.280+0300][3579309][gc] Allocation Stall (Notification Thread) 341,626ms
[2026-06-05T09:02:26.280+0300][3579309][gc] Allocation Stall (C1 CompilerThread0) 354,013ms
[2026-06-05T09:02:26.280+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 309,230ms
[2026-06-05T09:02:26.280+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 309,642ms
[2026-06-05T09:02:26.280+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 309,167ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 309,678ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 309,193ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 309,378ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 305,021ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 309,496ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 0,540ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-5) 0,552ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 310,547ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 305,218ms
[2026-06-05T09:02:26.281+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 310,848ms
[2026-06-05T09:02:26.282+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 311,573ms
[2026-06-05T09:02:26.284+0300][3579309][gc] Allocation Stall (parallel-8) 28,451ms
[2026-06-05T09:02:26.284+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 28,244ms
[2026-06-05T09:02:26.290+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 2,306ms
[2026-06-05T09:02:26.332+0300][3579309][gc] Allocation Stall (parallel-8) 43,162ms
[2026-06-05T09:02:26.332+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 40,411ms
[2026-06-05T09:02:26.345+0300][3579309][gc] GC(36) Minor Collection (Allocation Stall) 30720M(100%)->30704M(100%) 0,406s
[2026-06-05T09:02:26.357+0300][3579309][gc] GC(37) Minor Collection (Allocation Stall)
[2026-06-05T09:02:26.402+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 45,232ms
[2026-06-05T09:02:26.403+0300][3579309][gc] Allocation Stall (parallel-8) 45,928ms
[2026-06-05T09:02:26.403+0300][3579309][gc] Allocation Stall (pool-16-thread-1) 25,038ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 22,219ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 21,034ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 19,344ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 22,189ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 22,573ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 19,907ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 14,319ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 13,957ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 14,738ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 20,058ms
[2026-06-05T09:02:26.404+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 21,913ms
[2026-06-05T09:02:26.405+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 1,637ms
[2026-06-05T09:02:26.405+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 15,805ms
[2026-06-05T09:02:26.406+0300][3579309][gc] GC(37) Minor Collection (Allocation Stall) 30720M(100%)->30704M(100%) 0,049s
[2026-06-05T09:02:26.406+0300][3579309][gc] GC(38) Minor Collection (Allocation Stall)
[2026-06-05T09:02:26.465+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 48,413ms
[2026-06-05T09:02:26.465+0300][3579309][gc] Allocation Stall (httpclient-dispatch-12) 47,625ms
[2026-06-05T09:02:26.465+0300][3579309][gc] Allocation Stall (httpclient-dispatch-15) 47,701ms
[2026-06-05T09:02:26.465+0300][3579309][gc] Allocation Stall (parallel-8) 61,637ms
[2026-06-05T09:02:26.465+0300][3579309][gc] Allocation Stall (httpclient-dispatch-19) 47,625ms
[2026-06-05T09:02:26.465+0300][3579309][gc] Allocation Stall (httpclient-dispatch-10) 47,918ms
[2026-06-05T09:02:26.466+0300][3579309][gc] Allocation Stall (httpclient-dispatch-1) 47,697ms
[2026-06-05T09:02:26.466+0300][3579309][gc] Allocation Stall (httpclient-dispatch-20) 47,639ms
[2026-06-05T09:02:26.466+0300][3579309][gc] Allocation Stall (httpclient-dispatch-7) 47,628ms
[2026-06-05T09:02:26.466+0300][3579309][gc] Allocation Stall (httpclient-dispatch-3) 47,698ms
[2026-06-05T09:02:26.469+0300][3579309][gc] GC(38) Minor Collection (Allocation Stall) 30704M(100%)->30720M(100%) 0,063s
[2026-06-05T09:02:26.469+0300][3579309][gc] GC(39) Minor Collection (Allocation Stall)
[2026-06-05T09:02:26.504+0300][3579309][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 84,601ms
[2026-06-05T09:02:26.504+0300][3579309][gc] Allocation Stall (httpclient-dispatch-14) 86,393ms
[2026-06-05T09:02:26.504+0300][3579309][gc] Allocation Stall (httpclient-dispatch-4) 84,851ms
[2026-06-05T09:02:26.504+0300][3579309][gc] Allocation Stall (httpclient-dispatch-6) 86,378ms
[2026-06-05T09:02:26.504+0300][3579309][gc] Allocation Stall (httpclient-dispatch-13) 84,254ms
[2026-06-05T09:02:26.504+0300][3579309][gc] Allocation Stall (httpclient-dispatch-5) 84,819ms
[2026-06-05T09:02:26.504+0300][3579309][gc] Allocation Stall (httpclient-dispatch-8) 86,328ms
[2026-06-05T09:02:26.504+0300][3579309][gc] Allocation Stall (httpclient-dispatch-2) 84,836ms
[2026-06-05T09:02:26.505+0300][3579309][gc] Allocation Stall (httpclient-dispatch-16) 84,746ms
[2026-06-05T09:02:26.505+0300][3579309][gc] Allocation Stall (httpclient-dispatch-17) 84,817ms
[2026-06-05T09:02:26.505+0300][3579309][gc] Allocation Stall (httpclient-dispatch-18) 84,581ms
[2026-06-05T09:02:26.506+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-1) 0,120ms
[2026-06-05T09:02:26.506+0300][3579309][gc] Allocation Stall (httpclient-dispatch-11) 84,852ms
[2026-06-05T09:02:26.506+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 78,074ms
[2026-06-05T09:02:26.506+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 76,880ms
[2026-06-05T09:02:26.506+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 0,101ms
[2026-06-05T09:02:26.507+0300][3579309][gc] GC(39) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,038s
[2026-06-05T09:02:26.825+0300][3579309][gc] GC(40) Minor Collection (Allocation Stall)
[2026-06-05T09:02:26.860+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 430,281ms
[2026-06-05T09:02:26.860+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 420,795ms
[2026-06-05T09:02:26.860+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 430,216ms
[2026-06-05T09:02:26.860+0300][3579309][gc] Allocation Stall (parallel-8) 393,014ms
[2026-06-05T09:02:26.863+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 395,385ms
[2026-06-05T09:02:26.863+0300][3579309][gc] Allocation Stall (oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser) 393,850ms
[2026-06-05T09:02:26.863+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 358,891ms
[2026-06-05T09:02:26.864+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 359,006ms
[2026-06-05T09:02:26.864+0300][3579309][gc] Relocation Stall (oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser) 0,117ms
[2026-06-05T09:02:26.864+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 358,966ms
[2026-06-05T09:02:26.864+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 38,674ms
[2026-06-05T09:02:26.866+0300][3579309][gc] GC(40) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,041s
[2026-06-05T09:02:26.866+0300][3579309][gc] GC(41) Minor Collection (Allocation Stall)
[2026-06-05T09:02:26.902+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 62,249ms
[2026-06-05T09:02:26.903+0300][3579309][gc] GC(41) Minor Collection (Allocation Stall) 30720M(100%)->30712M(100%) 0,036s
[2026-06-05T09:02:26.965+0300][3579309][gc] GC(42) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.004+0300][3579309][gc] GC(42) Minor Collection (Allocation Stall) 30720M(100%)->30716M(100%) 0,039s
[2026-06-05T09:02:27.061+0300][3579309][gc] GC(43) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.092+0300][3579309][gc] GC(43) Minor Collection (Allocation Stall) 30720M(100%)->30718M(100%) 0,030s
[2026-06-05T09:02:27.418+0300][3579309][gc] GC(44) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.448+0300][3579309][gc] GC(44) Minor Collection (Allocation Stall) 30720M(100%)->30718M(100%) 0,031s
[2026-06-05T09:02:27.467+0300][3579309][gc] GC(45) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.498+0300][3579309][gc] GC(45) Minor Collection (Allocation Stall) 30720M(100%)->30718M(100%) 0,031s
[2026-06-05T09:02:27.506+0300][3579309][gc] GC(46) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.537+0300][3579309][gc] GC(46) Minor Collection (Allocation Stall) 30720M(100%)->30718M(100%) 0,031s
[2026-06-05T09:02:27.678+0300][3579309][gc] GC(47) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.710+0300][3579309][gc] GC(47) Minor Collection (Allocation Stall) 30720M(100%)->30718M(100%) 0,031s
[2026-06-05T09:02:27.751+0300][3579309][gc] GC(48) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.781+0300][3579309][gc] GC(48) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,030s
[2026-06-05T09:02:27.790+0300][3579309][gc] GC(49) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.822+0300][3579309][gc] GC(49) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,031s
[2026-06-05T09:02:27.832+0300][3579309][gc] GC(50) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.862+0300][3579309][gc] GC(50) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,030s
[2026-06-05T09:02:27.891+0300][3579309][gc] GC(51) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.921+0300][3579309][gc] GC(51) Minor Collection (Allocation Stall) 30720M(100%)->30718M(100%) 0,030s
[2026-06-05T09:02:27.940+0300][3579309][gc] GC(52) Minor Collection (Allocation Stall)
[2026-06-05T09:02:27.970+0300][3579309][gc] GC(52) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,030s
[2026-06-05T09:02:28.005+0300][3579309][gc] GC(53) Minor Collection (Allocation Stall)
[2026-06-05T09:02:28.034+0300][3579309][gc] GC(53) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,029s
[2026-06-05T09:02:28.227+0300][3579309][gc] GC(54) Minor Collection (Allocation Stall)
[2026-06-05T09:02:28.257+0300][3579309][gc] GC(54) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,030s
[2026-06-05T09:02:28.262+0300][3579309][gc] GC(55) Minor Collection (Allocation Stall)
[2026-06-05T09:02:28.292+0300][3579309][gc] GC(55) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,030s
[2026-06-05T09:02:28.467+0300][3579309][gc] GC(56) Minor Collection (Allocation Stall)
[2026-06-05T09:02:28.497+0300][3579309][gc] GC(56) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,030s
[2026-06-05T09:02:29.025+0300][3579309][gc] GC(57) Minor Collection (Allocation Stall)
[2026-06-05T09:02:29.056+0300][3579309][gc] GC(57) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,031s
[2026-06-05T09:02:29.679+0300][3579309][gc] GC(58) Minor Collection (Allocation Stall)
[2026-06-05T09:02:29.709+0300][3579309][gc] GC(58) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,030s
[2026-06-05T09:02:29.922+0300][3579309][gc] GC(59) Minor Collection (Allocation Stall)
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 1983,493ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/ui/default) 2244,841ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/workflow) 244,619ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (Notification Thread) 3056,920ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (httpclient-dispatch-11) 2415,909ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (httpclient-dispatch-15) 1455,624ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (parallel-2) 1661,121ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (qtp881248216-332) 2172,121ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 898,280ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 2862,621ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-9) 2503,832ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-1) 2457,661ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-10) 2457,738ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (Reference Handler) 1,426ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 2863,578ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (commons-pool-evictor) 2068,201ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 1918,060ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (Common-Cleaner) 1,075ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 3058,654ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 2863,198ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 3053,333ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 2959,770ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-6) 2419,629ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-19) 2457,758ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-16) 2419,621ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-20) 2457,663ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-14) 2419,218ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-3) 2457,663ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (parallel-7) 2091,238ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (fileinstall-/opt/ankey/idm/conf) 2132,761ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-12) 2457,857ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (httpclient-dispatch-4) 2419,843ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (qtp881248216-156-acceptor-0@4a87020a-0.0.0.0:8080@2fc102ab{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 2172,307ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (httpclient-dispatch-8) 2419,992ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (httpclient-dispatch-2) 2419,877ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (httpclient-dispatch-7) 2458,513ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-1-1) 2032,241ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (parallel-1) 1698,779ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (httpclient-dispatch-17) 2420,013ms
[2026-06-05T09:02:29.923+0300][3579309][gc] Allocation Stall (parallel-8) 3061,656ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (httpclient-dispatch-13) 2419,715ms
[2026-06-05T09:02:29.925+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 3059,621ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (fileinstall-./load) 2508,233ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 3054,478ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (httpclient-dispatch-18) 2419,947ms
[2026-06-05T09:02:29.926+0300][3579309][gc] Allocation Stall (httpclient-dispatch-5) 2419,874ms
[2026-06-05T09:02:29.956+0300][3579309][gc] GC(59) Minor Collection (Allocation Stall) 30720M(100%)->30534M(99%) 0,035s
[2026-06-05T09:02:30.024+0300][3579309][gc] GC(60) Minor Collection (Allocation Stall)
[2026-06-05T09:02:30.086+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 62,174ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 62,512ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 56,857ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-4) 0,199ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 62,915ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 0,035ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 56,904ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 57,329ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (parallel-5) 57,888ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 57,512ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-1) 0,195ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 61,345ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 61,231ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 61,643ms
[2026-06-05T09:02:30.092+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 66,652ms
[2026-06-05T09:02:30.093+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 60,993ms
[2026-06-05T09:02:30.093+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 5,592ms
[2026-06-05T09:02:30.093+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-5) 1,000ms
[2026-06-05T09:02:30.093+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 67,260ms
[2026-06-05T09:02:30.093+0300][3579309][gc] Relocation Stall (parallel-8) 1,272ms
[2026-06-05T09:02:30.093+0300][3579309][gc] Allocation Stall (parallel-8) 68,966ms
[2026-06-05T09:02:30.096+0300][3579309][gc] GC(60) Minor Collection (Allocation Stall) 30720M(100%)->30616M(100%) 0,072s
[2026-06-05T09:02:30.188+0300][3579309][gc] GC(61) Minor Collection (Allocation Stall)
[2026-06-05T09:02:30.264+0300][3579309][gc] Allocation Stall (parallel-8) 76,460ms
[2026-06-05T09:02:30.264+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 76,632ms
[2026-06-05T09:02:30.264+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 71,644ms
[2026-06-05T09:02:30.264+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 71,373ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 72,822ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 72,860ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 72,723ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 72,439ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 72,884ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 72,067ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 72,006ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 72,331ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (parallel-8) 1,556ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 72,391ms
[2026-06-05T09:02:30.266+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 72,422ms
[2026-06-05T09:02:30.268+0300][3579309][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 1,837ms
[2026-06-05T09:02:30.269+0300][3579309][gc] GC(61) Minor Collection (Allocation Stall) 30720M(100%)->30694M(100%) 0,081s
[2026-06-05T09:02:30.280+0300][3579309][gc] GC(62) Minor Collection (Allocation Stall)
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 12,006ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 11,180ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 11,931ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Relocation Stall (pool-13-thread-1) 0,088ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 11,480ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 11,920ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 10,794ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 11,110ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 11,034ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 12,009ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 11,369ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 11,348ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 11,937ms
[2026-06-05T09:02:30.379+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 99,349ms
[2026-06-05T09:02:30.380+0300][3579309][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 1,464ms
[2026-06-05T09:02:30.380+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-6) 0,921ms
[2026-06-05T09:02:30.382+0300][3579309][gc] Relocation Stall (parallel-8) 2,897ms
[2026-06-05T09:02:30.382+0300][3579309][gc] Allocation Stall (parallel-8) 101,615ms
[2026-06-05T09:02:30.385+0300][3579309][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 4,905ms
[2026-06-05T09:02:30.385+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-1) 5,449ms
[2026-06-05T09:02:30.428+0300][3579309][gc] GC(62) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,148s
[2026-06-05T09:02:30.428+0300][3579309][gc] GC(63) Minor Collection (Allocation Stall)
[2026-06-05T09:02:30.492+0300][3579309][gc] Allocation Stall (parallel-8) 109,855ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (Connector-Scheduler-2fc102ab-1) 603,460ms
[2026-06-05T09:02:31.033+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-3) 599,230ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (Notification Thread) 598,812ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-6-1) 101,019ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (qtp881248216-157-acceptor-1@1274689e-0.0.0.0:8080@2fc102ab{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}) 604,226ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-6) 592,249ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 591,004ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 592,500ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 592,390ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 541,588ms
[2026-06-05T09:02:31.034+0300][3579309][gc] Allocation Stall (Grizzly-IdleTimeoutFilter-IdleCheck) 108,592ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 545,993ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 542,112ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (parallel-8) 524,256ms
[2026-06-05T09:02:31.034+0300][3579309][gc] Allocation Stall (httpclient-dispatch-11) 109,067ms
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (httpclient-dispatch-20) 23,818ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 633,502ms
[2026-06-05T09:02:31.027+0300][3579309][gc] Allocation Stall (httpclient-dispatch-15) 102,281ms
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-4-1) 95,406ms
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (httpclient-dispatch-16) 23,811ms
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (httpclient-dispatch-8) 24,158ms
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (AsyncHttpClient-timer-1-1) 43,535ms
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (httpclient-dispatch-3) 24,151ms
[2026-06-05T09:02:31.034+0300][3579309][gc] GC(63) Minor Collection (Allocation Stall) 30720M(100%)->30720M(100%) 0,606s
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (httpclient-dispatch-4) 24,126ms
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (httpclient-dispatch-17) 24,001ms
[2026-06-05T09:02:31.035+0300][3579309][gc] Allocation Stall (httpclient-dispatch-12) 24,005ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (httpclient-dispatch-2) 24,150ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (httpclient-dispatch-7) 24,215ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (httpclient-dispatch-13) 24,256ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (httpclient-dispatch-9) 24,022ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (httpclient-dispatch-6) 24,083ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (httpclient-dispatch-10) 24,145ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (httpclient-dispatch-1) 24,211ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (C1 CompilerThread1) 23,391ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (qtp881248216-332) 8,129ms
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser) 0,835ms
[2026-06-05T09:02:31.036+0300][3579309][gc] GC(64) Minor Collection (Allocation Stall)
[2026-06-05T09:02:31.036+0300][3579309][gc] Allocation Stall (qtp881248216-336) 8,051ms
[2026-06-05T09:02:31.076+0300][3579309][gc] Allocation Stall (oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser) 38,869ms
[2026-06-05T09:02:31.076+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-4) 38,879ms
[2026-06-05T09:02:31.078+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 41,338ms
[2026-06-05T09:02:31.078+0300][3579309][gc] Allocation Stall (qtp881248216-336) 41,292ms
[2026-06-05T09:02:31.078+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 41,348ms
[2026-06-05T09:02:31.078+0300][3579309][gc] Relocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 0,039ms
[2026-06-05T09:02:31.078+0300][3579309][gc] Relocation Stall (reactive-kafka-ankey_default-2) 0,146ms
[2026-06-05T09:02:31.078+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-5) 41,346ms
[2026-06-05T09:02:31.079+0300][3579309][gc] Allocation Stall (qtp881248216-332) 41,637ms
[2026-06-05T09:02:31.078+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-2) 41,613ms
[2026-06-05T09:02:31.079+0300][3579309][gc] Allocation Stall (reactive-kafka-ankey_default-1) 41,682ms
[2026-06-05T09:02:31.078+0300][3579309][gc] Allocation Stall (kafka-coordinator-heartbeat-thread | ankey_default) 41,415ms
[2026-06-05T09:02:31.079+0300][3579309][gc] Relocation Stall (pool-13-thread-1) 0,656ms
[2026-06-05T09:02:31.079+0300][3579309][gc] Allocation Stall (pool-13-thread-1) 41,604ms
[2026-06-05T09:02:31.079+0300][3579309][gc] GC(64) Minor Collection (Allocation Stall) 30720M(100%)->30714M(100%) 0,043s
[2026-06-05T09:02:31.079+0300][3579309][gc] GC(65) Minor Collection (Allocation Stall)
[2026-06-05T09:02:31.115+0300][3579309][gc] GC(65) Minor Collection (Allocation Stall) 30716M(100%)->30702M(100%) 0,036s
Диагноз окончательный и неопровержимый.
Что показывает лог
После увеличения кучи до 30 ГБ (30720M) картина та же:
```
30720M(100%)->30720M(100%)
```
Куча снова забита под завязку. ZGC не может освободить память, потому что все 30 ГБ заняты живыми объектами.
Пиковые значения:
· 28514M(93%)->29304M(95%) — после сборки память выросла
· 30720M(100%) — достигнут потолок
Что это значит
Вашему IDM нужно больше 30 ГБ heap только для live data. Это не всплеск, не утечка — это рабочий объём приложения в вашей конфигурации.
Учитывая, что на стенде всего 47 ГБ ОЗУ:
· 30 ГБ у IDM
· 5 ГБ у OpenSearch
· 4 ГБ у Kafka
· Остаётся 8 ГБ на ОС и файловый кэш — это опасно мало
Почему CPU простаивает
При 100% заполнении кучи ZGC даже не пытается работать — он в сталлах, треды приложения заблокированы. CPU не загружен, потому что приложение не исполняется, оно ждёт память.
Ваши варианты (выбирайте)
1. Увеличить кучу до 35–38 ГБ (рискованно)
· Останется 5–9 ГБ на всё остальное
· OpenSearch и Kafka начнут падать или тормозить
· Требует тестирования
2. Разнести сервисы (рекомендуется)
· IDM на отдельную машину с 48+ ГБ RAM
· Или хотя бы вынести Kafka на другой стенд
3. Сократить рабочий объём IDM (самое правильное)
Вот что жрёт память судя по логу:
· reactive-kafka — несколько консьюмеров, каждый держит буферы
· httpclient-dispatch — много параллельных запросов (видно 20+ тредов)
· fileinstall — видимо, горячая деплоя конфигов
Что сделать:
```properties
# В конфиге Kafka consumer
max.poll.records=50 (было 500?)
fetch.max.bytes=5242880 (5MB вместо 50MB)
# В настройках IDM кэшей
cache.max.entries=10000
session.timeout.minutes=30
```
4. Сделать heap dump и найти корень
```bash
jcmd $(pgrep -f idm) GC.heap_info
jmap -dump:live,format=b,file=/tmp/idm.hprof $(pgrep -f idm)
```
(осторожно — дамп будет 30+ ГБ)
Немедленное действие
Покажите конфиги IDM (особенно launcher.json или security.properties) — найду параметры кэшей и Kafka, которые можно урезать.
Но главное: ваш стенд физически не тянет IDM с таким объёмом данных. Нужно либо больше RAM, либо разносить сервисы.