1. Kill the hung process properly

bash

sudo systemctl stop elasticsearch.service
# If it hangs:
sudo systemctl kill elasticsearch.service --signal=SIGKILL
# Or manually:
sudo kill -9 791
sudo systemctl reset-failed elasticsearch.service

2. Check Elasticsearch logs for confirmation

bash

sudo tail -n 100 /var/log/elasticsearch/elk-lab.log
sudo tail -n 100 /var/log/elasticsearch/gc.log 2>/dev/null

Look for:

• heap size [%s], compressed ordinary object pointers [%s] confirming 2g heap
• Long GC pauses
• ClusterApplierService taking a long time
• Shard recovery messages stuck on initialization

3. Increase systemd startup timeout (immediate workaround)

Create an override so systemd waits longer than 15 minutes:

bash

sudo mkdir -p /etc/systemd/system/elasticsearch.service.d/
sudo tee /etc/systemd/system/elasticsearch.service.d/timeout.conf << 'EOF'
[Service]
TimeoutStartSec=600
EOF
sudo systemctl daemon-reload

4. Reduce JVM heap size (strongly recommended)

With 2.5 GiB total RAM, a 2 GiB heap is unsustainable. Reduce it to 1 GiB or 1.5 GiB max:

bash

sudo nano /etc/elasticsearch/jvm.options

Change:

code

-Xms1g
-Xmx1g

Do not go above 50% of physical RAM (ideally keep it under 1g here since K3s, Jenkins, and Logstash are also running).

5. Restart and monitor

bash

sudo systemctl start elasticsearch.service
sleep 10
sudo systemctl status elasticsearch.service --no-pager

Watch the log in another terminal:

bash

sudo tail -f /var/log/elasticsearch/elk-lab.log

---

Long-Term Recommendation

Running Elasticsearch + Logstash + Kibana + K3s + Jenkins on a 2.5 GiB VM is not viable. Even if you reduce the heap, you will keep hitting memory pressure and swap thrashing.

• Preferred: Move Elasticsearch to a dedicated node with at least 4–8 GiB RAM, or
• Alternative: Add more RAM to master3 and separate the ELK and K3s workloads onto different nodes.