Vision db no data

Document created by viavisupport on Mar 20, 2020
Version 1Show Document
  • View in full screen mode

Solution

 

Vision no data being consumed at ingest

 

df -k showed /tmp partition full

 

on investigation full with hs_err, these are java dumps due to kafka process crashing.

 

also in /var/log/messages we see

 

Mar 19 09:30:17 db1 kafka-server-start.sh: 2020-03-19 09:30:17,173 WARN KAFKA.Log: Found a corrupted index file due to requirement failed: Corrupt index found, index file (/bigdee/opt/data/kafka/logs/kafka_lte_segments-1/00000000005655889921.index) has non-zero size but the last offset is 5655889921 which is no larger than the base offset5655889921.}. deleting /bigdee/opt/data/kafka/logs/kafka_lte_segments-1/00000000005655889921.timeindex, /bigdee/opt/data/kafka/logs/kafka_lte_segments-1/00000000005655889921.index, and /bigdee/opt/data/kafka/logs/kafka_lte_segments-1/00000000005655889921.txnindex and rebuilding index...

Mar 19 09:30:17 db1 kafka-server-start.sh: 2020-03-19 09:30:17,194 WARN KAFKA.Log: Found a corrupted index file due to requirement failed: Corrupt index found, index file (/bigdee/opt/data/kafka/logs/__consumer_offsets-37/00000000000000000388.index) has non-zero size but the last offset is 388 which is no larger than the base offset 388.}. deleting /bigdee/opt/data/kafka/logs/__consumer_offsets-37/00000000000000000388.timeindex, /bigdee/opt/data/kafka/logs/__consumer_offsets-37/00000000000000000388.index, and /bigdee/opt/data/kafka/logs/__consumer_offsets-37/00000000000000000388.txnindex and rebuilding index...

Mar 19 09:30:17 db1 kafka-server-start.sh: 2020-03-19 09:30:17,212 WARN KAFKA.Log: Found a corrupted index file due to requirement failed: Corrupt index found, index file (/bigdee/opt/data/kafka/logs/kafka_umts_segments-5/00000000001474807281.index) has non-zero size but the last offset is 1474807281 which is no larger than the base offset 1474807281.}. deleting /bigdee/opt/data/kafka/logs/kafka_umts_segments-5/00000000001474807281.timeindex, /bigdee/opt/data/kafka/logs/kafka_umts_segments-5/00000000001474807281.index, and /bigdee/opt/data/kafka/logs/kafka_umts_segments-5/00000000001474807281.txnindex and rebuilding index...

 

Problem

go to the vision cluster with the depot then run the repair on bigdata

 

cd /visionreleases/Vision//ansible

 

sh deployer.sh   "bigdata" -e repair=true

Software/Hardware Version

 

any with vision db

Ticket Number

 

C007364

Attachments

    Outcomes