注册 登录  
 加关注
   显示下一条  |  关闭
温馨提示!由于新浪微博认证机制调整,您的新浪微博帐号绑定已过期,请重新绑定!立即重新绑定新浪微博》  |  关闭

PostgreSQL research

公益是一辈子的事, I'm 德哥@Digoal, Just Do it!

 
 
 

日志

 
 

gmond configure  

2014-09-10 15:01:18|  分类: NOC |  标签: |举报 |字号 订阅

  下载LOFTER 我的照片书  |
ganglia的安装已经帮助请参考

首先回忆一下gmond的架构, gmond实际上是使用metric module收集信息, 通过通道发送, 另一方面, 监听并接收和聚合来自网络的gmond节点发送过来的metric信息.
下面这张图是多播场景.
gmond configure - 德哥@Digoal - PostgreSQL research

下面这张图也是多播场景, 只是配置了mute或者deaf, deaf只发不收, mute只收不发.
gmond configure - 德哥@Digoal - PostgreSQL research

下面这个图是单播场景.
gmond configure - 德哥@Digoal - PostgreSQL research

本文讲一下gmond的配置, 详细的请参考man gmond.conf
首先我们看看gmond的命令帮助.

[root@db-172-16-3-221 man]# gmond -h
gmond 3.6.0

The Ganglia Monitoring Daemon (gmond) listens to the cluster
message channel, stores the data in-memory and when requested
will output an XML description of the state of the cluster

Usage: gmond [OPTIONS]...

  -h, --help             Print help and exit
  -V, --version          Print version and exit
  -c, --conf=STRING      Location of gmond configuration file  
                           (default=`/opt/ganglia-core-3.6.0/etc/gmond.conf')
  -l, --location=STRING  Location of this host in the cluster 
                           'rack,rank,plane'.  (default=`0,0,0')
  -d, --debug=INT        Debug level. If greater than zero, daemon will stay in 
                           foreground.  (default=`0')
  -f, --foreground       Run in foreground (don't daemonize)  (default=off)
  -t, --default_config   Print the default configuration to stdout and exit  
                           (default=off)
  -m, --metrics          Print the list of metrics this gmond supports  
                           (default=off)
  -b, --bandwidth        Calculate minimum bandwidth use for configuration  
                           (default=off)
  -r, --convert=STRING   Convert a 2.5.x configuration file to the new 3.x 
                           format
  -p, --pid-file=STRING  Write process-id to file


注意几项 : 
产生一个默认的配置文件
  -t, --default_config   Print the default configuration to stdout and exit  
                           (default=off)
查看当前gmond core支持的metric
  -m, --metrics          Print the list of metrics this gmond supports  
                           (default=off)
评估当前gmond配置下的带宽需求
  -b, --bandwidth        Calculate minimum bandwidth use for configuration  
                           (default=off)

我们看一下3.6.0支持的metric, 配置文件不存在, 因此只输出了core支持的metric, 如果有配置文件的话, 会输出包含自定义module在内的所有metric.

[root@db-172-16-3-221 man]# gmond -m
Configuration file '/opt/ganglia-core-3.6.0/etc/gmond.conf' not found.
cpu_steal       cpu_steal (module cpu_module)
load_one        One minute load average (module load_module)
mem_sreclaimable        Amount of reclaimable slab memory (module mem_module)
os_release      Operating system release date (module sys_module)
mem_total       Total amount of memory displayed in KBs (module mem_module)
cpu_intr        cpu_intr (module cpu_module)
proc_run        Total number of running processes (module proc_module)
load_five       Five minute load average (module load_module)
disk_free       Total free disk space (module disk_module)
gexec           gexec available (module core_metrics)
mem_cached      Amount of cached memory (module mem_module)
mtu             Network maximum transmission unit (module sys_module)
cpu_sintr       cpu_sintr (module cpu_module)
pkts_in         Packets in per second (module net_module)
location        Location of the machine (module core_metrics)
bytes_in        Number of bytes in per second (module net_module)
swap_total      Total amount of swap space displayed in KBs (module mem_module)
bytes_out       Number of bytes out per second (module net_module)
load_fifteen    Fifteen minute load average (module load_module)
mem_free        Amount of available memory (module mem_module)
os_name         Operating system name (module sys_module)
boottime        The last time that the system was started (module sys_module)
cpu_idle        Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request (module cpu_module)
cpu_nice        Percentage of CPU utilization that occurred while executing at the user level with nice priority (module cpu_module)
cpu_user        Percentage of CPU utilization that occurred while executing at the user level (module cpu_module)
cpu_aidle       Percent of time since boot idle CPU (module cpu_module)
sys_clock       Time as reported by the system clock (module sys_module)
mem_buffers     Amount of buffered memory (module mem_module)
cpu_system      Percentage of CPU utilization that occurred while executing at the system level (module cpu_module)
part_max_used   Maximum percent used for all partitions (module disk_module)
disk_total      Total available disk space (module disk_module)
heartbeat       Last heartbeat (module core_metrics)
mem_shared      Amount of shared memory (module mem_module)
machine_type    System architecture (module sys_module)
cpu_wio         Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request (module cpu_module)
cpu_num         Total number of CPUs (module cpu_module)
proc_total      Total number of processes (module proc_module)
cpu_speed       CPU Speed in terms of MHz (module cpu_module)
pkts_out        Packets out per second (module net_module)
swap_free       Amount of available swap memory (module mem_module)


因为当前没有配置文件, 需要创建一个配置文件, 使用gmond -t
# gmond -t > /opt/ganglia-core-3.6.0/etc/gmond.conf
这样输出的配置文件可能不能完全覆盖目前所有的module, 例如 : 
mem_sreclaimable        Amount of reclaimable slab memory (module mem_module)
mtu             Network maximum transmission unit (module sys_module)
sys_clock       Time as reported by the system clock (module sys_module)
这几个就没有覆盖到.

简单的讲一下配置文件的含义 : 

[root@db-172-16-3-221 man]# less /opt/ganglia-core-3.6.0/etc/gmond.conf 
/* This configuration is as close to 2.5.x default behavior as possible
   The values closely match ./gmond/metric.h definitions in 2.5.x */
globals {    # 全局配置
  daemonize = yes   # 后台运行
  setuid = yes    # 启动gmond时, 设置gmond的运行uid
  user = nobody    # 以 nobody运行
  debug_level = 0   #  DEBUG级别
  max_udp_msg_len = 1472    # UDP包的长度, 注意, 如果UDP包要跨广域网传播或者VPN传播的话, 可能会导致封装后的大小超过MTU, 导致包丢失, 那么可以减小这个包的大小.
  mute = no   # 是否为哑巴, 只接收来自其他gmond的监控信息
  deaf = no    # 是否为聋子, 只发送监控信息
  allow_extra_data = yes   # 允许发送XML的EXTRA_DATA和EXTRA_ELEMENT部分信息, 如果完全自定义METRIC的话, 可以设置为FALSE, 那么将不发送XML的EXTRA_DATA和EXTRA_ELEMENT部分信息
  host_dmax = 86400   /*secs. Expires (removes from web interface) hosts in 1 day */  # 在指定时间内没有收到主机的任何报告时, 删除主机的metadata. 
  host_tmax = 20 /*secs */  # 在指定时间*4内没收到主机的任何报告, 超时.
  cleanup_threshold = 300 /*secs */    # 清除主机前, 等待的时间.
  gexec = no   #  是否宣告开启了gexecd的主机调用gexec.
  # By default gmond will use reverse DNS resolution when displaying your hostname
  # Uncommeting following value will override that value.
  # override_hostname = "mywebserver.domain.com"
  # If you are not using multicast this value should be set to something other than 0.
  # Otherwise if you restart aggregator gmond you will get empty graphs. 60 seconds is reasonable
  send_metadata_interval = 0 /*secs */       # 发送主机metadata给其他主机的时间间隔, 默认情况下只在第一个gmond包或有其他gmond在网络中被发现时发送.

}

/*
 * The cluster attributes specified will be used as part of the <CLUSTER>
 * tag that will wrap all hosts collected by this instance.
 */
cluster {  # 集群配置
  name = "unspecified"     # 集群名, 
  owner = "unspecified"    # 所有者
  latlong = "unspecified"    # 地理位置
  url = "unspecified"   # URL
}

#例如 : 
#         cluster {
#           name = "Millennium Cluster"
#           owner = "UC Berkeley CS Dept."
#           latlong = "N37.37 W122.23"
#           url = "http://www.millennium.berkeley.edu/"
#         }

/* The host section describes attributes of the host, like the location */
host {
  location = "unspecified"   # 主机在机房中的位置例如1,2,3,        The numbers represent Rack, Rank and Plane respectively.
}

/* Feel free to specify as many udp_send_channels as you like.  Gmond
   used to only support having a single channel */
udp_send_channel {    # 发送信息的通道, 可以配多个udp_send_channel group, 但是要么使用单播, 要么使用多播.
  # bind_hostname = yes  # Highly recommended, soon to be default. 建议使用YES, 那么将使用源地址解析HOSTNAME, 这个HOSTNAME会被用到XML里面.
                       # This option tells gmond to use a source address # 例如 解析出来的 HOST NAME="db-172-16-3-221.localdomain"
                       # that resolves to the machine's hostname.  Without
                       # this, the metrics may appear to come from any
                       # interface and the DNS names associated with
                       # those IPs will be used to create the RRDs.
  mcast_join = 239.2.11.71    # 多播地址
bind = 172.16.3.221 # 绑定地址, You can use the bind attribute to bind to a particular local address to be used as the source for the multicast
# packets sent or let gmond resolve the default hostname if bind_hostname = yes.
  port = 8649   # 端口
  ttl = 1   # 生存时间, 如果要跨路由器的话, 需要配置大的ttl, 否则会不可达.
  # buffer = 10485760   /*        If you have a large system with lots of metrics, you might experience UDP drops. 
       This happens when gmond is not able to process the UDP fast enough from the network. 
       In this case you might consider changing your setup into a more distributed setup using aggregator gmond hosts.
      Alternatively you can choose to create a bigger receive buffer: */ 同时需要配置系统内核# sysctl -w net.core.rmem_max=10485760
}

/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
mcast_join = 239.2.11.71
port = 8649
bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
# buffer = 10485760
}

/* You can specify as many tcp_accept_channels as you like to share
   an xml description of the state of the cluster */
tcp_accept_channel {    
  port = 8649
  # If you want to gzip XML output
  gzip_output = no
}

#模块
/* Each metrics module that is referenced by gmond must be specified and
   loaded. If the module has been statically linked with gmond, it does
   not require a load path. However all dynamically loadable modules must
   include a load path. */
modules {
  module {
    name = "core_metrics"
  }
  module {
    name = "cpu_module"
    path = "modcpu.so"
  }
  module {
    name = "disk_module"
    path = "moddisk.so"
  }
  module {
    name = "load_module"
    path = "modload.so"
  }
  module {
    name = "mem_module"
    path = "modmem.so"
  }
  module {
    name = "net_module"
    path = "modnet.so"
  }
  module {
    name = "proc_module"
    path = "modproc.so"
  }
  module {
    name = "sys_module"
    path = "modsys.so"
  }
}

/* This collection group will cause a heartbeat (or beacon) to be sent every
   20 seconds.  In the heartbeat is the GMOND_STARTED data which expresses
   the age of the running gmond. */
collection_group {
  collect_once = yes     # 表示只在gmond启动时采集一次, 表示这是一个稳定性指标, 一次采集就可以了
  time_threshold = 20   # 间隔20秒发送一次
  metric {
    name = "heartbeat"   # 填写gmond -m输出的metric name.
    title = "HeartBeat"   # 可以不写
  }
}

/* This collection group will send general info about this host every
   1200 secs.
   This information doesn't change between reboots and is only collected
   once. */
collection_group {
  collect_once = yes    # 同样只采集一次, 因为这些都是稳定指标
  time_threshold = 1200  # 1200秒发送一次
  metric {
    name = "cpu_num"
    title = "CPU Count"
  }
  metric {
    name = "cpu_speed"
    title = "CPU Speed"
  }
  metric {
    name = "mem_total"
    title = "Memory Total"
  }
  /* Should this be here? Swap can be added/removed between reboots. */
  metric {
    name = "swap_total"
    title = "Swap Space Total"
  }
  metric {
    name = "boottime"
    title = "Last Boot Time"
  }
  metric {
    name = "machine_type"
    title = "Machine Type"
  }
  metric {
    name = "os_name"
    title = "Operating System"
  }
  metric {
    name = "os_release"
    title = "Operating System Release"
  }
  metric {
    name = "location"
    title = "Location"
  }
}

/* This collection group will send the status of gexecd for this host
   every 300 secs.*/
/* Unlike 2.5.x the default behavior is to report gexecd OFF. */
collection_group {
  collect_once = yes
  time_threshold = 300
  metric {
    name = "gexec"
    title = "Gexec Status"
  }
}

/* This collection group will collect the CPU status info every 20 secs.
   The time threshold is set to 90 seconds.  In honesty, this
   time_threshold could be set significantly higher to reduce
   unneccessary  network chatter. */
collection_group {
  collect_every = 20
  time_threshold = 90
  /* CPU status */
  metric {
    name = "cpu_user"
    value_threshold = "1.0"   # 上次采集的数据和当前数据如果相差超过1.0, 则将当前的整个collection group发送.
    title = "CPU User"    #        It’s important to note that all metrics in a collection group are sent even when only a single                                               #        value_threshold is surpassed.
  }
  metric {
    name = "cpu_system"
    value_threshold = "1.0"
    title = "CPU System"
  }
  metric {
    name = "cpu_idle"
    value_threshold = "5.0"
    title = "CPU Idle"
  }
  metric {
    name = "cpu_nice"
    value_threshold = "1.0"
    title = "CPU Nice"
  }
  metric {
    name = "cpu_aidle"
    value_threshold = "5.0"
    title = "CPU aidle"
  }
  metric {
    name = "cpu_wio"
    value_threshold = "1.0"
    title = "CPU wio"
  }
  metric {
    name = "cpu_steal"
    value_threshold = "1.0"
    title = "CPU steal"
  }
  /* The next two metrics are optional if you want more detail...
     ... since they are accounted for in cpu_system.
  metric {
    name = "cpu_intr"
    value_threshold = "1.0"
    title = "CPU intr"
  }
  metric {
    name = "cpu_sintr"
    value_threshold = "1.0"
    title = "CPU sintr"
  }
  */
}

collection_group {
  collect_every = 20
  time_threshold = 90
  /* Load Averages */
  metric {
    name = "load_one"
    value_threshold = "1.0"
    title = "One Minute Load Average"
  }
  metric {
    name = "load_five"
    value_threshold = "1.0"
    title = "Five Minute Load Average"
  }
  metric {
    name = "load_fifteen"
    value_threshold = "1.0"
    title = "Fifteen Minute Load Average"
  }
}

/* This group collects the number of running and total processes */
collection_group {
  collect_every = 80
  time_threshold = 950
  metric {
    name = "proc_run"
    value_threshold = "1.0"
    title = "Total Running Processes"
  }
  metric {
    name = "proc_total"
    value_threshold = "1.0"
    title = "Total Processes"
  }
}

/* This collection group grabs the volatile memory metrics every 40 secs and
   sends them at least every 180 secs.  This time_threshold can be increased
   significantly to reduce unneeded network traffic. */
collection_group {
  collect_every = 40
  time_threshold = 180
  metric {
    name = "mem_free"
    value_threshold = "1024.0"
    title = "Free Memory"
  }
  metric {
    name = "mem_shared"
    value_threshold = "1024.0"
    title = "Shared Memory"
  }
  metric {
    name = "mem_buffers"
    value_threshold = "1024.0"
    title = "Memory Buffers"
  }
  metric {
    name = "mem_cached"
    value_threshold = "1024.0"
    title = "Cached Memory"
  }
  metric {
    name = "swap_free"
    value_threshold = "1024.0"
    title = "Free Swap Space"
  }
}

collection_group {
  collect_every = 40
  time_threshold = 300
  metric {
    name = "bytes_out"
    value_threshold = 4096
    title = "Bytes Sent"
  }
  metric {
    name = "bytes_in"
    value_threshold = 4096
    title = "Bytes Received"
  }
  metric {
    name = "pkts_in"
    value_threshold = 256
    title = "Packets Received"
  }
  metric {
    name = "pkts_out"
    value_threshold = 256
    title = "Packets Sent"
  }
}

/* Different than 2.5.x default since the old config made no sense */
collection_group {
  collect_every = 1800
  time_threshold = 3600
  metric {
    name = "disk_total"
    value_threshold = 1.0
    title = "Total Disk Space"
  }
}

collection_group {
  collect_every = 40
  time_threshold = 180
  metric {
    name = "disk_free"
    value_threshold = 1.0
    title = "Disk Space Available"
  }
  metric {
    name = "part_max_used"
    value_threshold = 1.0
    title = "Maximum Disk Space Used"
  }
}
# 可以包含其他配置文件
include ("/opt/ganglia-core-3.6.0/etc/conf.d/*.conf")

其他配置, 用于udp_recv_channel and tcp_accept_channel的访问控制 : 

ACCESS CONTROL
       The udp_recv_channel and tcp_accept_channel directives can contain an Access Control List (ACL).  This ACL
       allows you to specify exactly which hosts gmond process data from.

       An example of an acl entry looks like

         acl {
           default = "deny"
           access {
             ip = 192.168.0.4
             mask = 32
             action = "allow"
           }
         }


因为gmond一次传输的DATA可能包含大量的数据, 所以为了防止UDP一个包放不下, 如果要加大udp buffer. 同时需要配置内核 : 

# sysctl -w net.core.rmem_max=10485760
# vi /etc/sysctl.conf
net.core.rmem_max=10485760


启动gmond : 

# gmond -c /opt/ganglia-core-3.6.0/etc/gmond.conf 

[root@db-172-16-3-221 man]# netstat -anp|grep gmond
tcp        0      0 0.0.0.0:8649                0.0.0.0:*                   LISTEN      4462/gmond          
udp        0      0 239.2.11.71:8649            0.0.0.0:*                               4462/gmond          
udp        0      0 172.16.3.221:28390          239.2.11.71:8649            ESTABLISHED 4462/gmond 


启动后, 直接telnet 8649端口, 可以看到XML组织的监控数据和metadata.

[root@db-172-16-3-221 man]# telnet 127.0.0.1 8649
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"?>
<!DOCTYPE GANGLIA_XML [
   <!ELEMENT GANGLIA_XML (GRID|CLUSTER|HOST)*>
      <!ATTLIST GANGLIA_XML VERSION CDATA #REQUIRED>
      <!ATTLIST GANGLIA_XML SOURCE CDATA #REQUIRED>
   <!ELEMENT GRID (CLUSTER | GRID | HOSTS | METRICS)*>
      <!ATTLIST GRID NAME CDATA #REQUIRED>
      <!ATTLIST GRID AUTHORITY CDATA #REQUIRED>
      <!ATTLIST GRID LOCALTIME CDATA #IMPLIED>
   <!ELEMENT CLUSTER (HOST | HOSTS | METRICS)*>
      <!ATTLIST CLUSTER NAME CDATA #REQUIRED>
      <!ATTLIST CLUSTER OWNER CDATA #IMPLIED>
      <!ATTLIST CLUSTER LATLONG CDATA #IMPLIED>
      <!ATTLIST CLUSTER URL CDATA #IMPLIED>
      <!ATTLIST CLUSTER LOCALTIME CDATA #REQUIRED>
   <!ELEMENT HOST (METRIC)*>
      <!ATTLIST HOST NAME CDATA #REQUIRED>
      <!ATTLIST HOST IP CDATA #REQUIRED>
      <!ATTLIST HOST LOCATION CDATA #IMPLIED>
      <!ATTLIST HOST TAGS CDATA #IMPLIED>
      <!ATTLIST HOST REPORTED CDATA #REQUIRED>
      <!ATTLIST HOST TN CDATA #IMPLIED>
      <!ATTLIST HOST TMAX CDATA #IMPLIED>
      <!ATTLIST HOST DMAX CDATA #IMPLIED>
      <!ATTLIST HOST GMOND_STARTED CDATA #IMPLIED>
   <!ELEMENT METRIC (EXTRA_DATA*)>
      <!ATTLIST METRIC NAME CDATA #REQUIRED>
      <!ATTLIST METRIC VAL CDATA #REQUIRED>
      <!ATTLIST METRIC TYPE (string | int8 | uint8 | int16 | uint16 | int32 | uint32 | float | double | timestamp) #REQUIRED>
      <!ATTLIST METRIC UNITS CDATA #IMPLIED>
      <!ATTLIST METRIC TN CDATA #IMPLIED>
      <!ATTLIST METRIC TMAX CDATA #IMPLIED>
      <!ATTLIST METRIC DMAX CDATA #IMPLIED>
      <!ATTLIST METRIC SLOPE (zero | positive | negative | both | unspecified) #IMPLIED>
      <!ATTLIST METRIC SOURCE (gmond) 'gmond'>
   <!ELEMENT EXTRA_DATA (EXTRA_ELEMENT*)>
   <!ELEMENT EXTRA_ELEMENT EMPTY>
      <!ATTLIST EXTRA_ELEMENT NAME CDATA #REQUIRED>
      <!ATTLIST EXTRA_ELEMENT VAL CDATA #REQUIRED>
   <!ELEMENT HOSTS EMPTY>
      <!ATTLIST HOSTS UP CDATA #REQUIRED>
      <!ATTLIST HOSTS DOWN CDATA #REQUIRED>
      <!ATTLIST HOSTS SOURCE (gmond | gmetad) #REQUIRED>
   <!ELEMENT METRICS (EXTRA_DATA*)>
      <!ATTLIST METRICS NAME CDATA #REQUIRED>
      <!ATTLIST METRICS SUM CDATA #REQUIRED>
      <!ATTLIST METRICS NUM CDATA #REQUIRED>
      <!ATTLIST METRICS TYPE (string | int8 | uint8 | int16 | uint16 | int32 | uint32 | float | double | timestamp) #REQUIRED>
      <!ATTLIST METRICS UNITS CDATA #IMPLIED>
      <!ATTLIST METRICS SLOPE (zero | positive | negative | both | unspecified) #IMPLIED>
      <!ATTLIST METRICS SOURCE (gmond) 'gmond'>
]>
<GANGLIA_XML VERSION="3.6.0" SOURCE="gmond">
<CLUSTER NAME="test" LOCALTIME="1410332382" OWNER="digoal" LATLONG="111 122" URL="http://dba.sky-mobi.com">
<HOST NAME="db-172-16-3-221.localdomain" IP="172.16.3.221" TAGS="" REPORTED="1410332366" TN="15" TMAX="20" DMAX="86400" LOCATION="1,2,3" GMOND_STARTED="1410332246">
<METRIC NAME="mem_shared" VAL="0" TYPE="float" UNITS="KB" TN="15" TMAX="180" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="memory"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Amount of shared memory"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Shared Memory"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_wio" VAL="0.0" TYPE="float" UNITS="%" TN="45" TMAX="90" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU wio"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" TN="135" TMAX="1200" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="system"/>
<EXTRA_ELEMENT NAME="DESC" VAL="System architecture"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Machine Type"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="proc_total" VAL="1011" TYPE="uint32" UNITS=" " TN="55" TMAX="950" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="process"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Total number of processes"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Total Processes"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_num" VAL="48" TYPE="uint16" UNITS="CPUs" TN="135" TMAX="1200" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Total number of CPUs"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU Count"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_speed" VAL="1995" TYPE="uint32" UNITS="MHz" TN="135" TMAX="1200" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="CPU Speed in terms of MHz"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU Speed"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="pkts_out" VAL="0.00" TYPE="float" UNITS="packets/sec" TN="135" TMAX="300" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="network"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Packets out per second"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Packets Sent"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="swap_free" VAL="1048568" TYPE="float" UNITS="KB" TN="15" TMAX="180" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="memory"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Amount of available swap memory"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Free Swap Space"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_steal" VAL="0.0" TYPE="float" UNITS="%" TN="45" TMAX="90" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="cpu_steal"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU steal"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="load_one" VAL="0.00" TYPE="float" UNITS=" " TN="45" TMAX="70" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="load"/>
<EXTRA_ELEMENT NAME="DESC" VAL="One minute load average"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="One Minute Load Average"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="mem_total" VAL="99045568" TYPE="float" UNITS="KB" TN="135" TMAX="1200" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="memory"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Total amount of memory displayed in KBs"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Memory Total"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="os_release" VAL="2.6.32-431.el6.x86_64" TYPE="string" UNITS="" TN="135" TMAX="1200" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="system"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Operating system release date"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Operating System Release"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="proc_run" VAL="0" TYPE="uint32" UNITS=" " TN="55" TMAX="950" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="process"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Total number of running processes"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Total Running Processes"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="load_five" VAL="0.00" TYPE="float" UNITS=" " TN="45" TMAX="325" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="load"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Five minute load average"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Five Minute Load Average"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="gexec" VAL="OFF" TYPE="string" UNITS="" TN="135" TMAX="300" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="core"/>
<EXTRA_ELEMENT NAME="DESC" VAL="gexec available"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Gexec Status"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="disk_free" VAL="3209.367" TYPE="double" UNITS="GB" TN="135" TMAX="180" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="disk"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Total free disk space"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Disk Space Available"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="mem_cached" VAL="112696" TYPE="float" UNITS="KB" TN="15" TMAX="180" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="memory"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Amount of cached memory"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Cached Memory"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="pkts_in" VAL="0.00" TYPE="float" UNITS="packets/sec" TN="135" TMAX="300" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="network"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Packets in per second"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Packets Received"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="bytes_in" VAL="0.00" TYPE="float" UNITS="bytes/sec" TN="135" TMAX="300" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="network"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Number of bytes in per second"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Bytes Received"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="bytes_out" VAL="0.00" TYPE="float" UNITS="bytes/sec" TN="135" TMAX="300" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="network"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Number of bytes out per second"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Bytes Sent"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="swap_total" VAL="1048568" TYPE="float" UNITS="KB" TN="135" TMAX="1200" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="memory"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Total amount of swap space displayed in KBs"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Swap Space Total"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="mem_free" VAL="97487952" TYPE="float" UNITS="KB" TN="15" TMAX="180" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="memory"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Amount of available memory"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Free Memory"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="load_fifteen" VAL="0.00" TYPE="float" UNITS=" " TN="45" TMAX="950" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="load"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Fifteen minute load average"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Fifteen Minute Load Average"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="135" TMAX="1200" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="system"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Operating system name"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Operating System"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="boottime" VAL="1410314161" TYPE="uint32" UNITS="s" TN="135" TMAX="1200" DMAX="0" SLOPE="zero">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="system"/>
<EXTRA_ELEMENT NAME="DESC" VAL="The last time that the system was started"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Last Boot Time"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_idle" VAL="100.0" TYPE="float" UNITS="%" TN="45" TMAX="90" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU Idle"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_user" VAL="0.0" TYPE="float" UNITS="%" TN="45" TMAX="90" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Percentage of CPU utilization that occurred while executing at the user level"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU User"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_nice" VAL="0.0" TYPE="float" UNITS="%" TN="45" TMAX="90" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Percentage of CPU utilization that occurred while executing at the user level with nice priority"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU Nice"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_aidle" VAL="0.0" TYPE="float" UNITS="%" TN="45" TMAX="3800" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Percent of time since boot idle CPU"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU aidle"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="mem_buffers" VAL="15628" TYPE="float" UNITS="KB" TN="15" TMAX="180" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="memory"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Amount of buffered memory"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Memory Buffers"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="cpu_system" VAL="0.0" TYPE="float" UNITS="%" TN="45" TMAX="90" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="cpu"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Percentage of CPU utilization that occurred while executing at the system level"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="CPU System"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="part_max_used" VAL="30.6" TYPE="float" UNITS="%" TN="135" TMAX="180" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="disk"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Maximum percent used for all partitions"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Maximum Disk Space Used"/>
</EXTRA_DATA>
</METRIC>
<METRIC NAME="disk_total" VAL="3862.720" TYPE="double" UNITS="GB" TN="135" TMAX="1200" DMAX="0" SLOPE="both">
<EXTRA_DATA>
<EXTRA_ELEMENT NAME="GROUP" VAL="disk"/>
<EXTRA_ELEMENT NAME="DESC" VAL="Total available disk space"/>
<EXTRA_ELEMENT NAME="TITLE" VAL="Total Disk Space"/>
</EXTRA_DATA>
</METRIC>
</HOST>
</CLUSTER>
</GANGLIA_XML>
Connection closed by foreign host.


[参考]
3. man gmond.conf
4. 最终的gmond.conf配置

[root@db-172-16-3-221 etc]# cat gmond.conf
/* This configuration is as close to 2.5.x default behavior as possible
   The values closely match ./gmond/metric.h definitions in 2.5.x */
globals {
  daemonize = yes
  setuid = yes
  user = nobody
  debug_level = 0
  max_udp_msg_len = 1472
  mute = no
  deaf = no
  allow_extra_data = yes
  host_dmax = 86400 /*secs. Expires (removes from web interface) hosts in 1 day */
  host_tmax = 20 /*secs */
  cleanup_threshold = 300 /*secs */
  gexec = no
  # By default gmond will use reverse DNS resolution when displaying your hostname
  # Uncommeting following value will override that value.
  # override_hostname = "mywebserver.domain.com"
  # If you are not using multicast this value should be set to something other than 0.
  # Otherwise if you restart aggregator gmond you will get empty graphs. 60 seconds is reasonable
  send_metadata_interval = 0 /*secs */

}

/*
 * The cluster attributes specified will be used as part of the <CLUSTER>
 * tag that will wrap all hosts collected by this instance.
 */
cluster {
  name = "test"
  owner = "digoal"
  latlong = "111 122"
  url = "http://dba.sky-mobi.com"
}

/* The host section describes attributes of the host, like the location */
host {
  location = "1,2,3"
}

/* Feel free to specify as many udp_send_channels as you like.  Gmond
   used to only support having a single channel */
udp_send_channel {
  #bind_hostname = yes # Highly recommended, soon to be default.
                       # This option tells gmond to use a source address
                       # that resolves to the machine's hostname.  Without
                       # this, the metrics may appear to come from any
                       # interface and the DNS names associated with
                       # those IPs will be used to create the RRDs.
  mcast_join = 239.2.11.71
  port = 8649
  ttl = 10
}

/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
  mcast_join = 239.2.11.71
  port = 8649
  bind = 239.2.11.71
  retry_bind = true
  # Size of the UDP buffer. If you are handling lots of metrics you really
  # should bump it up to e.g. 10MB or even higher.
  buffer = 10485760
}

/* You can specify as many tcp_accept_channels as you like to share
   an xml description of the state of the cluster */
tcp_accept_channel {
  port = 8649
  # If you want to gzip XML output
  gzip_output = no
}

/* Channel to receive sFlow datagrams */
#udp_recv_channel {
#  port = 6343
#}

/* Optional sFlow settings */
#sflow {
# udp_port = 6343
# accept_vm_metrics = yes
# accept_jvm_metrics = yes
# multiple_jvm_instances = no
# accept_http_metrics = yes
# multiple_http_instances = no
# accept_memcache_metrics = yes
# multiple_memcache_instances = no
#}

/* Each metrics module that is referenced by gmond must be specified and
   loaded. If the module has been statically linked with gmond, it does
   not require a load path. However all dynamically loadable modules must
   include a load path. */
modules {
  module {
    name = "core_metrics"
  }
  module {
    name = "cpu_module"
    path = "modcpu.so"
  }
  module {
    name = "disk_module"
    path = "moddisk.so"
  }
  module {
    name = "load_module"
    path = "modload.so"
  }
  module {
    name = "mem_module"
    path = "modmem.so"
  }
  module {
    name = "net_module"
    path = "modnet.so"
  }
  module {
    name = "proc_module"
    path = "modproc.so"
  }
  module {
    name = "sys_module"
    path = "modsys.so"
  }
}

/* The old internal 2.5.x metric array has been replaced by the following
   collection_group directives.  What follows is the default behavior for
   collecting and sending metrics that is as close to 2.5.x behavior as
   possible. */

/* This collection group will cause a heartbeat (or beacon) to be sent every
   20 seconds.  In the heartbeat is the GMOND_STARTED data which expresses
   the age of the running gmond. */
collection_group {
  collect_once = yes
  time_threshold = 20
  metric {
    name = "heartbeat"
  }
}

/* This collection group will send general info about this host every
   1200 secs.
   This information doesn't change between reboots and is only collected
   once. */
collection_group {
  collect_once = yes
  time_threshold = 1200
  metric {
    name = "mtu"
    title = "MTU"
  }
  metric {
    name = "cpu_num"
    title = "CPU Count"
  }
  metric {
    name = "cpu_speed"
    title = "CPU Speed"
  }
  metric {
    name = "mem_total"
    title = "Memory Total"
  }
  /* Should this be here? Swap can be added/removed between reboots. */
  metric {
    name = "swap_total"
    title = "Swap Space Total"
  }
  metric {
    name = "boottime"
    title = "Last Boot Time"
  }
  metric {
    name = "machine_type"
    title = "Machine Type"
  }
  metric {
    name = "os_name"
    title = "Operating System"
  }
  metric {
    name = "os_release"
    title = "Operating System Release"
  }
  metric {
    name = "location"
    title = "Location"
  }
}

/* This collection group will send the status of gexecd for this host
   every 300 secs.*/
/* Unlike 2.5.x the default behavior is to report gexecd OFF. */
collection_group {
  collect_once = yes
  time_threshold = 300
  metric {
    name = "gexec"
    title = "Gexec Status"
  }
}

/* This collection group will collect the CPU status info every 20 secs.
   The time threshold is set to 90 seconds.  In honesty, this
   time_threshold could be set significantly higher to reduce
   unneccessary  network chatter. */
collection_group {
  collect_every = 20
  time_threshold = 90
  /* sys clock */
  metric {
    name = "sys_clock"
    title = "System clock"
  }
  metric {
    name = "mem_sreclaimable"
    value_threshold = "1.0"
    title = "Reclaimable slab memory"
  }
  /* CPU status */
  metric {
    name = "cpu_user"
    value_threshold = "1.0"
    title = "CPU User"
  }
  metric {
    name = "cpu_system"
    value_threshold = "1.0"
    title = "CPU System"
  }
  metric {
    name = "cpu_idle"
    value_threshold = "5.0"
    title = "CPU Idle"
  }
  metric {
    name = "cpu_nice"
    value_threshold = "1.0"
    title = "CPU Nice"
  }
  metric {
    name = "cpu_aidle"
    value_threshold = "5.0"
    title = "CPU aidle"
  }
  metric {
    name = "cpu_wio"
    value_threshold = "1.0"
    title = "CPU wio"
  }
  metric {
    name = "cpu_steal"
    value_threshold = "1.0"
    title = "CPU steal"
  }
  /* The next two metrics are optional if you want more detail...
     ... since they are accounted for in cpu_system.
  metric {
    name = "cpu_intr"
    value_threshold = "1.0"
    title = "CPU intr"
  }
  metric {
    name = "cpu_sintr"
    value_threshold = "1.0"
    title = "CPU sintr"
  }
  */
}

collection_group {
  collect_every = 20
  time_threshold = 90
  /* Load Averages */
  metric {
    name = "load_one"
    value_threshold = "1.0"
    title = "One Minute Load Average"
  }
  metric {
    name = "load_five"
    value_threshold = "1.0"
    title = "Five Minute Load Average"
  }
  metric {
    name = "load_fifteen"
    value_threshold = "1.0"
    title = "Fifteen Minute Load Average"
  }
}

/* This group collects the number of running and total processes */
collection_group {
  collect_every = 80
  time_threshold = 950
  metric {
    name = "proc_run"
    value_threshold = "1.0"
    title = "Total Running Processes"
  }
  metric {
    name = "proc_total"
    value_threshold = "1.0"
    title = "Total Processes"
  }
}

/* This collection group grabs the volatile memory metrics every 40 secs and
   sends them at least every 180 secs.  This time_threshold can be increased
   significantly to reduce unneeded network traffic. */
collection_group {
  collect_every = 40
  time_threshold = 180
  metric {
    name = "mem_free"
    value_threshold = "1024.0"
    title = "Free Memory"
  }
  metric {
    name = "mem_shared"
    value_threshold = "1024.0"
    title = "Shared Memory"
  }
  metric {
    name = "mem_buffers"
    value_threshold = "1024.0"
    title = "Memory Buffers"
  }
  metric {
    name = "mem_cached"
    value_threshold = "1024.0"
    title = "Cached Memory"
  }
  metric {
    name = "swap_free"
    value_threshold = "1024.0"
    title = "Free Swap Space"
  }
}

collection_group {
  collect_every = 40
  time_threshold = 300
  metric {
    name = "bytes_out"
    value_threshold = 4096
    title = "Bytes Sent"
  }
  metric {
    name = "bytes_in"
    value_threshold = 4096
    title = "Bytes Received"
  }
  metric {
    name = "pkts_in"
    value_threshold = 256
    title = "Packets Received"
  }
  metric {
    name = "pkts_out"
    value_threshold = 256
    title = "Packets Sent"
  }
}

/* Different than 2.5.x default since the old config made no sense */
collection_group {
  collect_every = 1800
  time_threshold = 3600
  metric {
    name = "disk_total"
    value_threshold = 1.0
    title = "Total Disk Space"
  }
}

collection_group {
  collect_every = 40
  time_threshold = 180
  metric {
    name = "disk_free"
    value_threshold = 1.0
    title = "Disk Space Available"
  }
  metric {
    name = "part_max_used"
    value_threshold = 1.0
    title = "Maximum Disk Space Used"
  }
}

include ("/opt/ganglia-core-3.6.0/etc/conf.d/*.conf")


Flag Counter
  评论这张
 
阅读(1534)| 评论(0)
推荐 转载

历史上的今天

在LOFTER的更多文章

评论

<#--最新日志,群博日志--> <#--推荐日志--> <#--引用记录--> <#--博主推荐--> <#--随机阅读--> <#--首页推荐--> <#--历史上的今天--> <#--被推荐日志--> <#--上一篇,下一篇--> <#-- 热度 --> <#-- 网易新闻广告 --> <#--右边模块结构--> <#--评论模块结构--> <#--引用模块结构--> <#--博主发起的投票-->
 
 
 
 
 
 
 
 
 
 
 
 
 
 

页脚

网易公司版权所有 ©1997-2016