Skip to main content
Version: v1.5.x

Monitoring Hadoop

Collect and monitor general performance metrics for the JVM virtual machine in Hadoop.

Protocol used: JMX

Pre-monitoring steps

You need to enable JMX service in the Hadoop application before monitoring. HertzBeat uses the JMX protocol to collect performance metrics from Hadoop's JVM.

Steps to enable JMX protocol in the Hadoop application

Add JVM parameters when the application starts. ⚠️Note that you can customize the exposed port and external IP.

  • 1.Enter the hadoop-env.sh configuration file and enter the following command in the terminal:
vi $HADOOP_HOME/etc/hadoop/hadoop-env.sh
  • 2.Add the following parameters, where port is the number of the custom-exposed port
export HADOOP_OPTS= "$HADOOP_OPTS
-Djava.rmi.server.hostname=对外ip地址
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false "
  • 3.Save and exit, and then execute "start-all.sh" in the "$HADOOP_HOME/sbin" directory to restart the service.

Configuration parameter

Parameter nameParameter help description
Monitoring HostMonitored IPV4, IPV6 or domain name. Note⚠️Without protocol header (eg: https://, http://)
Monitoring nameIdentify the name of this monitoring. The name needs to be unique
PortPort provided by JMX
UsernameJMX connection user name, optional
PasswordJMX connection password, optional
Collection intervalInterval time of monitor periodic data collection, unit: second, and the minimum interval that can be set is 30 seconds
Whether to detectWhether to detect and check the availability of monitoring before adding monitoring. Adding and modifying operations will continue only after the detection is successful
Description remarksFor more information about identifying and describing this monitoring, users can note information here

Collection Metrics

Metrics Set:memory_pool

Metric nameMetric unitMetric help description
namemetrics name
committedkbtotal size
initkbinit size
maxkbmax size
usedkbused size

Metrics Set:code_cache (Only Support JDK8)

Metric nameMetric unitMetric help description
committedkbtotal size
initkbinit size
maxkbmax size
usedkbused size

Metrics Set:class_loading

Metric nameMetric unitMetric help description
LoadedClassCountLoaded Class Count
TotalLoadedClassCountTotal Loaded Class Count
UnloadedClassCountUnloaded Class Count

Metrics Set:thread

Metric nameMetric unitMetric help description
TotalStartedThreadCountTotal Started Thread Count
ThreadCountThread Count
PeakThreadCountPeak Thread Count
DaemonThreadCountDaemon Thread Count
CurrentThreadUserTimemsCurrent Thread User Time
CurrentThreadCpuTimemsCurrent Thread Cpu Time