QNAP – HOW TO REPAIR RAID5 ARRAY WITH UNRECOVERED READ ERROR (URE) DURING REBUILD

February 24th, 2019 No comments

Tonight I experienced first hand a fear that most QNAP NAS owners have whilst rebuilding a broken drive in a RAID 5 array. You guessed it, the dreaded URE (UNRECOVERED READ ERROR).

# dmesg

[23298.622891] md: md0: recovery done.
[23298.628401] md: Recovering done: md0, degraded=1
[23299.732589] ata6.00: exception Emask 0x0 SAct 0x7fe SErr 0x0 action 0x0
[23299.737963] ata6.00: irq_stat 0x40000008
[23299.743250] ata6.00: failed command: READ FPDMA QUEUED
[23299.748477] ata6.00: cmd 60/00:08:84:d9:44/04:00:57:00:00/40 tag 1 ncq 524288 in
[23299.748481] res 41/40:00:f0:dc:44/00:00:57:00:00/40 Emask 0x409 (media error) <F>
[23299.759028] ata6.00: status: { DRDY ERR }
[23299.764316] ata6.00: error: { UNC }
[23299.782380] ata6.00: configured for UDMA/133
[23299.787537] ata6: EH complete
[23301.855721] ata6.00: exception Emask 0x0 SAct 0x7ff SErr 0x0 action 0x0
[23301.860887] ata6.00: irq_stat 0x40000008
[23301.866252] ata6.00: failed command: READ FPDMA QUEUED
[23301.871320] ata6.00: cmd 60/00:48:84:d9:44/04:00:57:00:00/40 tag 9 ncq 524288 in
[23301.871325] res 41/40:00:ef:dc:44/00:00:57:00:00/40 Emask 0x409 (media error) <F>
[23301.882098] ata6.00: status: { DRDY ERR }
[23301.887228] ata6.00: error: { UNC }
[23306.892039] ata6.00: qc timeout (cmd 0xec)
[23306.897030] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[23306.902001] ata6.00: revalidation failed (errno=-5)
[23306.906919] ata6: hard resetting link
[23307.371054] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[23307.388929] ata6.00: configured for UDMA/133
[23307.393885] ata6: EH complete
[23308.431943] ata6: log page 10h reported inactive tag 0
[23308.436777] ata6.00: exception Emask 0x1 SAct 0x7fe SErr 0x0 action 0x0
[23308.441659] ata6.00: irq_stat 0x40000008
[23308.446492] ata6.00: failed command: READ FPDMA QUEUED
[23308.451684] ata6.00: cmd 60/00:08:84:d9:44/04:00:57:00:00/40 tag 1 ncq 524288 in
[23308.451689] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.461723] ata6.00: status: { DRDY }
[23308.466716] ata6.00: failed command: READ FPDMA QUEUED
[23308.471658] ata6.00: cmd 60/00:10:84:dd:44/03:00:57:00:00/40 tag 2 ncq 393216 in
[23308.471661] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.481539] ata6.00: status: { DRDY }
[23308.486502] ata6.00: failed command: READ FPDMA QUEUED
[23308.491486] ata6.00: cmd 60/08:18:7a:6c:10/00:00:00:00:00/40 tag 3 ncq 4096 in
[23308.491490] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.501645] ata6.00: status: { DRDY }
[23308.506405] ata6.00: failed command: READ FPDMA QUEUED
[23308.511115] ata6.00: cmd 60/08:20:72:6c:10/00:00:00:00:00/40 tag 4 ncq 4096 in
[23308.511119] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.520405] ata6.00: status: { DRDY }
[23308.524995] ata6.00: failed command: READ FPDMA QUEUED
[23308.529534] ata6.00: cmd 60/08:28:6a:6c:10/00:00:00:00:00/40 tag 5 ncq 4096 in
[23308.529538] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.538586] ata6.00: status: { DRDY }
[23308.543067] ata6.00: failed command: READ FPDMA QUEUED
[23308.547584] ata6.00: cmd 60/08:30:62:6c:10/00:00:00:00:00/40 tag 6 ncq 4096 in
[23308.547588] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.556706] ata6.00: status: { DRDY }
[23308.561263] ata6.00: failed command: READ FPDMA QUEUED
[23308.565820] ata6.00: cmd 60/08:38:5a:6c:10/00:00:00:00:00/40 tag 7 ncq 4096 in
[23308.565825] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.574937] ata6.00: status: { DRDY }
[23308.579488] ata6.00: failed command: READ FPDMA QUEUED
[23308.584079] ata6.00: cmd 60/08:40:52:6c:10/00:00:00:00:00/40 tag 8 ncq 4096 in
[23308.584082] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.593387] ata6.00: status: { DRDY }
[23308.597981] ata6.00: failed command: READ FPDMA QUEUED
[23308.602812] ata6.00: cmd 60/08:48:4a:6c:10/00:00:00:00:00/40 tag 9 ncq 4096 in
[23308.602817] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.612334] ata6.00: status: { DRDY }
[23308.616907] ata6.00: failed command: READ FPDMA QUEUED
[23308.621563] ata6.00: cmd 60/08:50:42:6c:10/00:00:00:00:00/40 tag 10 ncq 4096 in
[23308.621567] res 40/00:54:42:6c:10/00:00:00:00:00/40 Emask 0x1 (device error)
[23308.631025] ata6.00: status: { DRDY }
[23308.648602] ata6.00: configured for UDMA/133
[23308.653389] ata6: EH complete
[23310.704908] ata6.00: exception Emask 0x0 SAct 0x3ff SErr 0x0 action 0x0
[23310.709669] ata6.00: irq_stat 0x40000008
[23310.714362] ata6.00: failed command: READ FPDMA QUEUED
[23310.719089] ata6.00: cmd 60/00:48:84:d9:44/04:00:57:00:00/40 tag 9 ncq 524288 in
[23310.719092] res 41/40:00:ef:dc:44/00:00:57:00:00/40 Emask 0x409 (media error) <F>
[23310.728625] ata6.00: status: { DRDY ERR }
[23310.733749] ata6.00: error: { UNC }
[23310.751334] ata6.00: configured for UDMA/133
[23310.756049] ata6: EH complete
[23312.863828] ata6.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[23312.868659] ata6.00: irq_stat 0x40000008
[23312.873247] ata6.00: failed command: READ FPDMA QUEUED
[23312.877828] ata6.00: cmd 60/00:00:84:d9:44/04:00:57:00:00/40 tag 0 ncq 524288 in
[23312.877832] res 41/40:00:ef:dc:44/00:00:57:00:00/40 Emask 0x409 (media error) <F>
[23312.887045] ata6.00: status: { DRDY ERR }
[23312.891653] ata6.00: error: { UNC }
[23312.909053] ata6.00: configured for UDMA/133
[23312.913591] ata6: EH complete
[23314.959481] ata6.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[23314.964017] ata6.00: irq_stat 0x40000008
[23314.968487] ata6.00: failed command: READ FPDMA QUEUED
[23314.972937] ata6.00: cmd 60/00:00:84:d9:44/04:00:57:00:00/40 tag 0 ncq 524288 in
[23314.972941] res 41/40:00:ef:dc:44/00:00:57:00:00/40 Emask 0x409 (media error) <F>
[23314.982330] ata6.00: status: { DRDY ERR }
[23314.986831] ata6.00: error: { UNC }
[23315.004157] ata6.00: configured for UDMA/133
[23315.008655] sd 5:0:0:0: [sda] Unhandled sense code
[23315.013027] sd 5:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[23315.017468] sd 5:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[23315.021882] Descriptor sense data with sense descriptors (in hex):
[23315.026341] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[23315.030854] 57 44 dc ef
[23315.035243] sd 5:0:0:0: [sda] Add. Sense: Unrecovered read error – auto reallocate failed
[23315.039729] sd 5:0:0:0: [sda] CDB: Read(10): 28 00 57 44 d9 84 00 04 00 00
[23315.044333] end_request: I/O error, dev sda, sector 1464130799
[23315.048820] md/raid:md0: read error not correctable (sector 1462010216 on sda3).
[23315.053422] raid5: some error occurred in a active device:0 of md0.
[23315.058027] raid5: Keep the raid device active in degraded mode but set readonly.
[23315.062687] md/raid:md0: read error not correctable (sector 1462010224 on sda3).
[23315.067315] raid5: some error occurred in a active device:0 of md0.
[25914.157523] md: ioctl lock interrupted, reason -4, cmd -2142762735

When you encounter a URE during a re-build, most of the advice on many of the forums suggest that the only course of action is to create a new array and restore your data from a backup.

As someone who does not give up that easy, I went looking for an alternative option and found my new best friend. ddrecue.

Using ddrescue, I was able to clone the disk that was having the URE (UNRECOVERED READ ERROR). To begin the process you must first identify the problem disk. Steps are:

  1. You need to ssh / putty into the NAS
  2. Run the command dmesg and locate a line similar to “read error not correctable (sector 1462010224 on sda3)“.
  3. Typically – sda3 = disk1, sdb3 = disk2, sdc3 = disk3, sdd3 = disk 4
  4. Eject the problem disk from the NAS
  5. Download and install SystemRescueCd on a USB stick – see http://www.system-rescue-cd.org/Installing-SystemRescueCd-on-a-USB-stick/
  6. On any computer, install the problem drive and the new replacement drive using SATA connectors.
  7. Configure the computer to boot from USB, plug in the SystemRescueCd on the USB stick and power-on
  8. At the command prompt use fdisk -l to identify your disks – typically /dev/sda and /dev/sdb
  9. To determine which is the source disk, you can use smartctl:|
    smartctl -a /dev/sda
    smartctl -a /dev/sdb
  10. You can also determine drive serial numbers with hdparm:
    hdparm -I /dev/sda
    hdparm -I /dev/sdb
  11. Once the source and destination disks are determined. Use ddrescue to copy the disk:
    ddrescue -f /dev/sda /dev/sdb
    Important Note: In this example /dev/sda is the source and /dev/sdb is the destination.
  12. Once the cloning of the disk completes, replace the problem disk with the new clone and start the RAID array rebuild again.
Categories: QNAP Tags:

Cowboy: An Agile Programming Methodology for a Solo Programmer

February 6th, 2015 No comments
Categories: Uncategorized Tags:

HowTo – Jenkins – Remove an Orphaned Workspace

December 14th, 2014 No comments

A couple of times I’ve been left with an orphaned workspace directory after deleting a project.

If you don’t the credentials to remove the directory in the operating system, try this Groovy script:

def command = """rm -rf /var/lib/jenkins/workspace/TheOrpanedWorkspace""" // Create the String
def proc = command.execute() // Call *execute* on the string
proc.waitFor()
print "done";

Categories: jenkins Tags:

Useful GIT Commands

September 14th, 2014 No comments

Clone a “bare” Repository
This command is useful to setup an intermediate repository to develop against instead of developing directly against github:

git clone --bare [remote github repo] [local repo]

Push all your branches in the “bare” repo to the github remote
This will push all your branches to the remote, and set-upstream tracking correctly for you:

git push --all -u

Categories: Uncategorized Tags:

HowTo – AIX – Emulate BASH Keystrokes

April 16th, 2014 No comments

If you are looking for an easy way to emulate BASH up / down arrow behavior as well as TAB completion of filenames and directories then this might be a solution for you:

  1. Change your default shell to ksh93 using the “chsh” command and select “/usr/bin/ksh93
  2. Edit you “.profile” file and add the following line:

    export ENV=$HOME/.kshrc

  3. Copy the contents of THIS FILE to $HOME/.kshrc
Categories: AIX Tags:

HowTo – XBMC: By-pass Shutdown Menu

March 10th, 2014 No comments

The following steps will allow you to by-pass the shutdown menu when you press the “power” button on the remote control and allow the box go directly into suspend mode (M3) or shutdown (M1):

  1. ssh into your Pivos box
  2. cp /usr/share/xbmc/system/keymaps/keyboard.xml to /root/.xbmc/userdata/keymaps/
  3. Edit /root/.xbmc/userdata/keymaps/keyboard.xml
  4. Change <power>ActivateWindow(shutdownmenu)</power> to <power>XBMC.Suspend()</power>
  5. Change <s>ActivateWindow(shutdownmenu)</s> to <s>XBMC.Suspend()</s>
  6. Reboot

Note: If your device is an M1 model, simply change XBMC.Suspend() to be XBMC.Shutdown() instead.

Categories: Uncategorized Tags:

HowTo: Setup WD TV Live Streaming in Australia

December 22nd, 2012 No comments
The following are a few simple steps to setup a WD TV Live Streaming Player to access apps (e.g. Netflix, HuluPlus etc.) that are blocked in some overseas countries such as Australia.

  1. Subscribe to unblock-us.com and following instructions described here: http://support.unblock-us.com/customer/portal/articles/291573
  2. Power off the WD TV Live using the remote power button (red). Wait 10 secs and power back on again.
  3. Go to the Setup screen
  4. Select system->Reset Location, a U.S. based timezone should be selected automatically if unblock-us is setup correctly.
  5. Once the location has been reset correctly, you can adjust the timezone setting to match your location.
  6. Go to Services
  7. Confirm that you have HuluPlus, Netflix etc.
Categories: WD TV Tags:

Android – How to send callers directly to voicemail

November 19th, 2012 No comments

For the last week or so I have been receiving annoying telemarketing calls. Today a found a real simple way to automatically re-direct these annoying calls to voicemail.

The following steps describe how to set this up in Android 4.1.2 (Jelly Bean).

  • Step 1– Navigate to the “People” application.

  • Step 2– Select the contact you wish to automatically re-direct to voicemail.

  • Step 3– Select the checkbox and exit.

Categories: Android Tags:

Script To Clear Event Logs

November 11th, 2012 No comments

Awesome script to clear all of your Windows 7 event logs. Wish I remembered where I sourced it from to give them credit.


@echo off
FOR /F "tokens=1,2*" %%V IN ('bcdedit') DO SET adminTest=%%V
IF (%adminTest%)==(Access) goto noAdmin
for /F "tokens=*" %%G in ('wevtutil.exe el') DO (call :do_clear "%%G")
echo.
echo Event Logs have been cleared! ^ goto theEnd
:do_clear
echo clearing %1
wevtutil.exe cl %1
goto :eof
:noAdmin
echo You must run this script as an Administrator!
echo ^ :theEnd
pause>NUL

Categories: Windows 7 Tags:

TextPad and Excel Slow To Open Files

November 11th, 2012 No comments

I recently discovered an issue on my Windows 7 x64 system where double clicking text files (associated with TextPad) and Excel files (associated with Excel) took over 30 seconds to open the file.

A few forums on the Web suggested that the problem was that a program on my system may be blocking DDE messaging. In my case it turned out that Acronis TrueImage 2012 was the issue.

After uninstalling this bad boy I found that my file open timings returned to normal.

Categories: Windows 7 Tags: