При нагрузке на жесткий диск система зависает. CENTOS 6.5

[klesk]

Приветствую, есть сервер для бэкапов. У него два харда в зеркале и один большой.
Я в линуксе не силен.
Сервер год работал без сбоев. Никакое ПО и железо не устанавливалось. Единственное возможно был выключен вручную.
Проверил температура процессора в норме. Сделал fsck -y ему.
При любой попытке что то записать на него по сети система виснет и помогает только отключение питания.
С чего начать?

[ruslandh]

С изучения. как он работает.
А потом переходить к логам его.

[klesk]

как кто работает?

[ruslandh]

сервер бэкапов

[klesk]

я описал как он работает, пока не пишешь на диск второй нормально работает, как пишешь - виснет.

[James Marsh]

Цитата klesk:
два харда в зеркале »
хард/материнковый/mdadm???

Цитата klesk:
пока не пишешь на диск второй нормально работает, как пишешь - виснет »
ХТО? ГДЕ??


#cat /proc/mdstat
#fdisk -l (минус эль)
#lsblk
#smartctl -a /dev/sdX (поочередно)
в студию

Без этого - гадание на кофейных зернах, собранными девственницами в полнолуние.

[klesk]

[root@centos1c ~]# cat /proc/mdstat
Personalities :
unused devices: <none>
[root@centos1c ~]# fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cc60e

Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 60667 486791168 8e Linux LVM

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cc60e

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sdb2 64 60667 486791168 8e Linux LVM

Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/ddf1_4c5349202020202080862925000000004711471100001450: 499.0 GB , 498999492608 bytes
255 heads, 63 sectors/track, 60666 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cc60e

Device Boot St art End Blocks Id System
/dev/mapper/ddf1_4c5349202020202080862925000000004711471100001450p1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/mapper/ddf1_4c5349202020202080862925000000004711471100001450p2 64 60667 486791168 8e Linux LVM

Disk /dev/mapper/ddf1_4c5349202020202080862925000000004711471100001450p1: 524 MB , 524288000 bytes
255 heads, 63 sectors/track, 63 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/ddf1_4c5349202020202080862925000000004711471100001450p2: 498.5 GB, 498474156032 bytes
255 heads, 63 sectors/track, 60602 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_centos1c-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_centos1c-lv_swap: 8086 MB, 8086618112 bytes
255 heads, 63 sectors/track, 983 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_centos1c-lv_home: 436.7 GB, 436698349568 bytes
255 heads, 63 sectors/track, 53092 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

[root@centos1c ~]#


[root@centos1c ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
└─ddf1_4c5349202020202080862925000000004711471100001450 (dm-0) 253:0 0 464.7G 0 dmraid
├─ddf1_4c5349202020202080862925000000004711471100001450p1 (dm-1) 253:1 0 500M 0 part /boot
└─ddf1_4c5349202020202080862925000000004711471100001450p2 (dm-2) 253:2 0 464.2G 0 part
├─vg_centos1c-lv_root (dm-3) 253:3 0 50G 0 lvm /
├─vg_centos1c-lv_swap (dm-4) 253:4 0 7.5G 0 lvm [SWAP]
└─vg_centos1c-lv_home (dm-5) 253:5 0 406.7G 0 lvm /home
sdb 8:16 0 465.8G 0 disk
└─ddf1_4c5349202020202080862925000000004711471100001450 (dm-0) 253:0 0 464.7G 0 dmraid
├─ddf1_4c5349202020202080862925000000004711471100001450p1 (dm-1) 253:1 0 500M 0 part /boot
└─ddf1_4c5349202020202080862925000000004711471100001450p2 (dm-2) 253:2 0 464.2G 0 part
├─vg_centos1c-lv_root (dm-3) 253:3 0 50G 0 lvm /
├─vg_centos1c-lv_swap (dm-4) 253:4 0 7.5G 0 lvm [SWAP]
└─vg_centos1c-lv_home (dm-5) 253:5 0 406.7G 0 lvm /home
sdc 8:32 0 1.8T 0 disk /backup
[root@centos1c ~]#

[root@centos1c ~]# smartctl -a /dev/sda
-bash: smartctl: command not found
[root@centos1c ~]# smartctl -a /dev/sdc
-bash: smartctl: command not found
[root@centos1c ~]#





[root@centos1c ~]# smartctl -a /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.11.2.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Blue Serial ATA
Device Model: WDC WD5000AAKX-00ERMA0
Serial Number: WD-WCC2EKF25198
LU WWN Device Id: 5 0014ee 2b373ef64
Firmware Version: 15.01H15
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Jan 14 22:31:16 2016 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 8460) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 86) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3037) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 141 141 021 Pre-fail Always - 3941
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 47
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 087 087 000 Old_age Always - 9504
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 47
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 36
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 12
194 Temperature_Celsius 0x0022 110 099 000 Old_age Always - 33
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@centos1c ~]#

----------------------------------------------------------------------------------------------------------------------------

[root@centos1c ~]# smartctl -a /dev/sdb
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.11.2.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Blue Serial ATA
Device Model: WDC WD5000AAKX-00ERMA0
Serial Number: WD-WCC2EKA86264
LU WWN Device Id: 5 0014ee 208c921ae
Firmware Version: 15.01H15
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Jan 14 22:33:29 2016 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 8700) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 88) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3037) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 253 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 142 142 021 Pre-fail Always - 3858
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 49
5 Reallocated_Sector_Ct 0x0033 186 186 140 Pre-fail Always - 294
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 087 087 000 Old_age Always - 9637
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 49
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 38
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 11
194 Temperature_Celsius 0x0022 111 100 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 186 186 000 Old_age Always - 14
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 1
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@centos1c ~]#

-----------------------------------------------------------------------------------------------------------------



[root@centos1c ~]# smartctl -a /dev/sdc
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.11.2.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model: WDC WD20EFRX-68EUZN0
Serial Number: WD-WMC4M1448305
LU WWN Device Id: 5 0014ee 65937d3a2
Firmware Version: 80.00A80
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ACS-2 (revision not indicated)
Local Time is: Thu Jan 14 22:34:21 2016 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (27360) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 276) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 172 169 021 Pre-fail Always - 4400
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 48
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 090 090 000 Old_age Always - 7956
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 48
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 37
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 1758
194 Temperature_Celsius 0x0022 120 110 000 Old_age Always - 27
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@centos1c ~]#

[zai]

У него два харда в зеркале »
[root@centos1c ~]# cat /proc/mdstat Personalities : unused devices: <none> »

[klesk]

я просто не помню, там софтовый интелловский рейд но для Centos вроде нашелся драйвер, не помню ( могу ошибаться). Если можно посмотреть, скажите как.

[James Marsh]

Судя по выводу lsblk и fdisk -l у Вас на двухтерабайтнике куда-то "улетел" раздел, бо в /backup примонтировано /dev/sdc, а не, к примеру, /dev/sdc1.

Выложите еще, пожалуйста:

#cat /etc/fstab

Глянем шо куды монтируется.

[klesk]

[root@centos1c ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Thu Apr 10 07:26:13 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_centos1c-lv_root / ext4 defaults 1 1
UUID=2471e1b9-514c-4cf4-a3ce-825e10fb25e4 /boot ext4 defaults 1 2
/dev/mapper/vg_centos1c-lv_home /home ext4 defaults 1 2
/dev/mapper/vg_centos1c-lv_swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
UUID=d960c53d-7554-4971-8c4d-b4368c6577fc /backup ext3 rw 0 0[root@centos1c ~]#

[James Marsh]

Я так понимаю, что файлов на бэкапе нема.

Тогда:
UUID=d960c53d-7554-4971-8c4d-b4368c6577fc /backup ext3 rw 0 0 »

Грохните эту запись в fstab'e
Создайте раздел /dev/sdc1
Форматните его в ту же ext3, если Вам хочется и ручками примонтируйте в /backup (кстати еще тот вопрос, кто Вас учил в корневом что-то создавать и туда монтировать.)
Проверьте и если все гут, то пропишите явно в fstab

/dev/sdc1 /backup ext3 rw 0 0

[klesk]

(кстати еще тот вопрос, кто Вас учил в корневом что-то создавать и туда монтировать.)
говорю я не очень разбираюсь, но год все работало гуд

нет, к сожалению, есть файлы и они нужны
http://s8.hostingkartinok.com/uploads/images/2016/01/8be8c20eff0b4698260cfb1bcc0c054e.png

[James Marsh]

Батенька, да у Вас там по ходу место свободное закончилось

[klesk]

почистил вроде нормально заработал, а чего так в линуксе отказ системы от нехватки места на несистемном диске нормальная ситуация?

[MakaBooka]

а чего так в линуксе отказ системы от нехватки места на несистемном диске нормальная ситуация? »
нет. но при неграмотно поставленом процессе возможно всякое: от забивания /tmp который часть рута до забивания свапа.
в идеале всё должно мониториться.