h1

Avamar version 5 SP2 Install Factory Bug

June 11, 2010

I was recently presented with a hurdle when trying to complete a fresh install of Avamar version 5 SP2 on a brand new fresh out of the box Avamar Multi-Node Data Store that had come from Manufacturing.

I received the following error a repeated number of times in the avqinstall.log. This error occured when the install process tried to start the gsan service:

“avqinstall: ERROR: the following lines from the output of

the preceding command indicated problems:

2010/06/10-14:36:54.86627 {0.0} [log-messages:108] ERROR: HARDWARE: Dec 14

15:59:14 localhost Server Administrator: Instrumentation Service EventID: 1012 IPMI status<span style="mso-spacerun:

yes”> Interface: OS

checkprimecomplete encountered a gsan error, stopped at

/usr/local/avamar/bin/start.dpn line 374.

avqinstall: ERROR: run command failed: ‘start.dpn –clean

–password=”8RttoTriz” –systemname=”UtilityNode” — “

Turns out there is a factory bug with the install for Avamar version 5 SP2.

The issue is caused by a bug fix (#18994) in v5.0 SP2 to correct the init script that generates the log scan configuration file used by Avamar server (GSAN) to filter unwanted Event IDs from the system log ‘/var/log/messages’. The fix opens the door to pass thru IPMI Event ID 1012, which is interpreted as an *error* in the log file, and causes a new install to fail. IPMI Event ID 1012 gets logged into the system log ‘/var/log/messages’ when the Dell OMSA gets installed & started, or restarted, or the system gets rebooted.

The workaround is to rename the system log by executing the command
‘mv /var/log/messages /var/log/messages.01’
on all nodes prior to an install, then perform the install without performing a reboot.

If you do not rename the system log, the install will most likely fail when getting to the part where the gsan service is started. If you have already encountered this error, in my case, the fix is to issue the mv /var/log/messages /var/log/messages.01′ command on all nodes and reinstall, however, before you attempt to install again, there are a few things you will need to do.
1. Verify if gsan’s are running after a failed avqinstall in multi-node by issuing command:

su – admin

ssh-agent bash

ssh-add ~/.ssh/dpnid

mapall –noerror ‘ps ax | grep gsan’

2. Kill the gsan process by issuing command:

mapall –noerror killall gsan

3. Then verify that they’re gone by issuing command:

mapall –noerror ‘ps ax | grep gsan’

4, Check status of all services on Utility Node by issuing command:

dpnctl status

5. If all is okay, install again using the following command:

avqinstall –norequire_clean_system –nounpack

*Please note that issuing the –norequire_clean_system will remove all backups and should be issued with extreme caution.

(The –nounpack will save sometime and avoid unpacking all of the install files all over again.)

I hope this helps save some headaches and time for someone out there.
Cheers-
HG


Advertisements

4 comments

  1. I enjoy your blog and your perspectives on the industry. I run the product management team for Avamar and we’d welcome the opportunity to get your opinions and input.

    If you’d like, drop me a line at greener_gail@emc.com


  2. With all of the Avamar nodes you’ve done, have you ever run into an error saying “segmentation fault” after re-kickstarting a Gen3 node? No one at EMC can figure this out, and I’ve seen 4-5 times now.


    • What version of Avamar 5 did you install? It wouldn’t be SP2 would it? I hear there is a bug in Avamar 5.0 SP2 server, bug#20004, whereby stripes get corrupted.
      Also, before you re-kickstarted it, did you wipe it clean? There should be a white paper on how to do it, if not, I believe I have it somewhere.


  3. Thanks for the fix. Interesting that I only had this problem one of the two multi-node servers at the site.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: