Troubleshooting Oracle Problems with 1.1.1 NBU backup

1. Create Debug folders under the following directories on database client:

All log directories should is created Under/usr/openv/netbackup/logs with 777 permissions.
/usr/openv/netbackup/logs/bphdb
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/dbclient

2. Attempt the backup or restore operation

3. Gather the appropriate debug information from the backup client

A. The content of all the error logs At/usr/openv/netbackup/logs
B. The backup script and the output of the backup script with exact error messages
C./usr/openv/netbackup/bp.conf and bp.conf at home directory of the Oracle OS user if exists.
D. The Oracle database version information
Sqlplus "/As SYSDBA"
Sql> select * from V$version;
Sql> select * from V$instance;
E. In case of the Oracle report critical errors (e.g. ORA-0600) or backup hang for a long time without response, we also need To collect the Oracle alert log at $ORACLE _base/admin/<sid>/bdump.

Common commands

///////////////////////////////////////////////
Windows OS

To adjust the log level of master server, the method is as follows:
Open the NBU console interface, Host Properties > Master Servers, right-click on the master server, select Logging, check the Enable robust Logging, set The global Logging level is 5.
Create a log directory
Install_path\netbackup\logs\bptm
Install_path\netbackup\logs\bpbrm
Install_path\netbackup\logs\vnetd
Install_path\netbackup\logs\bpcd
Install_path\netbackup\logs\bprd
install_path\netbackup\logs\bpdbm
========

To adjust the log level of media server, the method is as follows:
Open the NBU console interface, Host Properties > Media Servers, right media server, click Properties, select Logging, tick enable robust Logging, Set the global Logging level to 5.
Create a log directory
Install_path\netbackup\logs\bptm
Install_path\netbackup\logs\bpbrm
Install_path\netbackup\logs\vnetd
Install_path\netbackup\logs\bpcd
========

To adjust the log level for client server, the method is as follows:
On the client server, click "Start > All Programs >symantec netbackup>backup Archive & Restore",
At the bar console interface, File > Netbackup Client Properties > troubleshooting, set the general level to 2 and Verbose level to 5.
Create a log directory
Install_path\netbackup\logs\bpbkar
Install_path\netbackup\logs\bpfis
Install_path\netbackup\logs\vnetd
Install_path\netbackup\logs\bpcd
Install_path\netbackup\logs\bphdb
Install_path\netbackup\logs\dbclient
Install_path\netbackup\logs\bpdb2
Install_path\netbackup\logs\backint

///////////////////////////////////////////////
Unix/linux OS

To adjust the log level of master server, the method is as follows:
Add verbose = 5 to/usr/openv/netbackup/bp.conf file.
Create a log directory
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/vnetd
/usr/openv/netbackup/logs/bprd
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm
/usr/openv/netbackup/logs/bpdbm

Need to restart the NetBackup service,
/usr/openv/netbackup/bin/goodies/netbackup stop
/opt/vrtspbx/bin/vxpbx_exchanged stop
/usr/openv/netbackup/bin/bpps-x (except for the pbx_exchange process, no other NB process and mm process)
/opt/vrtspbx/bin/vxpbx_exchanged start
/usr/openv/netbackup/bin/goodies/netbackup start
========

To adjust the log level of media server, the method is as follows:
Add verbose = 5 to/usr/openv/netbackup/bp.conf file.
Create a log directory
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/vnetd
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm

Need to restart the NetBackup service,
/usr/openv/netbackup/bin/goodies/netbackup stop
/opt/vrtspbx/bin/vxpbx_exchanged stop
/usr/openv/netbackup/bin/bpps-x (except for the pbx_exchange process, no other NB process and mm process)
/opt/vrtspbx/bin/vxpbx_exchanged start
/usr/openv/netbackup/bin/goodies/netbackup start
========

To adjust the log level for client server, the method is as follows:
Add verbose = 5 to/usr/openv/netbackup/bp.conf file.
Create a log directory
/usr/openv/netbackup/logs/bpbkar
/usr/openv/netbackup/logs/bpfis
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bphdb
/usr/openv/netbackup/logs/dbclient
/usr/openv/netbackup/logs/bpdb2
/usr/openv/netbackup/logs/backint
/usr/openv/netbackup/logs/sybackup

///////////////////////////////////////////////
Socket Connection Failed problem

Follow these steps to test if the port of the NetBackup communication is OK,

On the master server,
Telnet Client_name 13724
Telnet Client_name 13782

Telnet Media_server_name 13724
Telnet Media_server_name 13782
Telnet Media_server_name 1556

On Media server,
Telnet Client_name 13724
Telnet Client_name 13782

Telnet Master_server_name 13724
Telnet Master_server_name 13782
Telnet Master_server_name 1556

On the client server,
Telnet Master_server_name 13724
Telnet Master_server_name 13782
Telnet Master_server_name 13720

Telnet Media_server_name 13724
Telnet Media_server_name 13782
Telnet Master_server_name 1556

If the port is not telnet, check:
Add the hostname and IP resolution of the peer server to the 1.hosts file correctly.
Ping Server_hostname
Ping server_ip_address

2. On the server to the peer, check that the Telnet port is listening.
Netstat-na
telnet localhost port_number

3. If Telnet localhost port_number is successful, check whether the firewall of the network or OS will port disable

///////////////////////////////////////////////
Unified logs,

====windows os=====
Adjust the relevant unified log level,
Install_path\netbackup\bin\vxlogcfg–a–p 51216–o 111–s debuglevel=6–s diagnosticlevel=6
Install_path\netbackup\bin\vxlogcfg–a–p 51216–o 116–s debuglevel=6–s diagnosticlevel=6
Install_path\netbackup\bin\vxlogcfg–a–p 51216–o 117–s debuglevel=6–s diagnosticlevel=6
Install_path\netbackup\bin\vxlogcfg–a–p 51216–o 118–s debuglevel=6–s diagnosticlevel=6

Collect log,
Install_path\netbackup\bin\vxlogview–p 51216–o 111–t 00:10:00–d all > C:\111.txt

====unix/linux os=====
Adjust the relevant unified log level,
/usr/openv/netbackup/bin/vxlogcfg–a–p 51216–o 111–s debuglevel=6–s diagnosticlevel=6

Collect log,
/usr/openv/netbackup/bin/vxlogview–p 51216–o 111–t 00:10:00–d all >/tmp/111.txt

//////////////////////////////////////////////

BMR Bmrsaveconfig

Please adjust the log level of the bmrsavecfg on this client.
Install_path\netbackup\bin\vxlogcfg–a–p 51216–o 121–s debuglevel=6–s diagnosticlevel=6

Run the command, if the command error, the error message will also be sent to me.
Install_path\netbackup\bin\bmrsavecfg–infoonly

Once the above command returns to the prompt, run the command to collect log
Install_path\netbackup\bin\vxlogview–p 51216–o 121–t 00:10:00–d all > C:\121.txt

Collect the following files on the host,
Install_path\netbackup\baremetal\client\data\bmrcli.xml & Bundle.dat

Lower the log level,
Install_path\netbackup\bin\vxlogcfg–a–p 51216–o 121–s debuglevel=1–s diagnosticlevel=1

=======================

/usr/openv/volmgr/bin/tpconfig-emm_dev_list >/tmp/emmdev_1.txt
/usr/openv/volmgr/bin/vmoprcmd >/tmp/ Vmoprcmd_1.txt
/usr/openv/netbackup/bin/admincmd/bppllist-allpolicies-l >/tmp/pol.txt
/usr/openv/ Netbackup/bin/admincmd/bpstulist-l >/tmp/stu.txt
/usr/openv/netbackup/bin/admincmd/bperror-u >/tmp/ Bperror.txt
/usr/openv/netbackup/bin/admincmd/bpdbjobs >/tmp/bpdbjobs.txt
/usr/openv/netbackup/bin/ Goodies/available_media >/tmp/am.txt
/usr/openv/netbackup/bin/admincmd/nbemmcmd-listmedia-allrecords >/ Tmp/emm_media.txt
/usr/openv/netbackup/bin/admincmd/nbemmcmd-listhosts-verbose >/tmp/nbemmcmd.txt

Adjust the log level of the NBEMM,NBRB,NBJM
/usr/openv/netbackup/bin/vxlogcfg-a-P 51216-o 111-s debuglevel=6-s diagnosticlevel=6
/usr/openv/netbackup/bin/vxlogcfg-a-P 51216-o 117-s debuglevel=6-s diagnosticlevel=6
/usr/openv/netbackup/bin/vxlogcfg-a-P 51216-o 118-s debuglevel=6-s diagnosticlevel=6

Simultaneously initiates 4 jobs, reproduces two runs & two queued phenomena (preferably this can last more than 3 minutes)
In the course of the continuation of this phenomenon,
/usr/openv/volmgr/bin/tpconfig-emm_dev_list >/tmp/emmdev_2.txt
/usr/openv/volmgr/bin/vmoprcmd >/tmp/vmoprcmd_2.txt

Collect log
/usr/openv/netbackup/bin/vxlogview-p 51216-o 111-t 00:10:00-d all >/tmp/111.txt
/usr/openv/netbackup/bin/vxlogview-p 51216-o 117-t 00:10:00-d all >/tmp/117.txt
/usr/openv/netbackup/bin/vxlogview-p 51216-o 118-t 00:10:00-d all >/tmp/118.txt

Send me the log and command output files above/tmp/.
Copy and paste the detailed status information from the latter two queued jobs to the TXT text file that you sent to me.
And tell me the name of the policy you initiated at the same time.

///////////////////

If I encounter a problem with the catalog backup failure, I will follow the procedure below
1. Confirm to the user whether the previous catalog backup was successful. If successful, you need to check the cause of this failure, or restart NBU, back up the catalog, see the results, if the new configuration, you need to check the configuration of the catalog policy.
2. Confirm that the backup catalog has several jobs and should be 4 job. Let the user send us the detailed status of the catalog failure job.
3. According to the status information to determine the initial problem, as in this case is 811

Check it first.
Install_path\netbackup\bin\admincmd\nbrbutil-dump > C:\nbrbdump.txt
Install_path\netbackup\bin\admincmd\bppllist-allpolicies-l > C:\pol.txt
Install_path\netbackup\bin\admincmd\bpstulist-l > C:\stu.txt
Install_path\netbackup\bin\admincmd\bperror-u > C:\bperror.txt
Install_path\netbackup\bin\admincmd\nbemmcmd-listhosts-verbose > C:\nbemmcmd.txt
Install_path\netbackup\bin\admincmd\bpminlicense-list_keys-verbose > C:\nbulic.txt
Install_path\netbackup\bin\admincmd\bpdbjobs > C:\job.txt

Install_path\volmgr\bin\vmglob-listall-b > C:\vmglob.txt
install_path\volmgr\bin\tpconfig-d > C:\tpconfig.txt
Install_path\volmgr\bin\vmoprcmd > C:\vmoprcmd.txt
Install_path\volmgr\bin\tpclean-l > C:\tpclean.txt

Collect related Log
Open the NBU console interface, Host Properties > Master Servers, right-click on the master server, select Logging, check the Enable robust Logging, set The global Logging level is 5.
Create a log directory
Install_path\netbackup\logs\bptm
Install_path\netbackup\logs\bpbrm

Restart the NBU service, re-launch the NBU catalog backup, if it fails, send us the log under Bptm and BPBRM.

Still fail, BPTM and BPBRM still have 811 error
Need to collect unified log

/usr/openv/netbackup/bin/vxlogcfg-a-P 51216-o 111-s debuglevel=6-s diagnosticlevel=6

Adjust NBEMM,NBRB,NBJM log level
/usr/openv/netbackup/bin/vxlogcfg-a-P 51216-o 111-s debuglevel=6-s DiagnosticLeve L=6
/usr/openv/netbackup/bin/vxlogcfg-a-P 51216-o 117-s debuglevel=6-s diagnosticlevel=6
/usr/openv/netbacku P/bin/vxlogcfg-a-P 51216-o 118-s debuglevel=6-s diagnosticlevel=6
Re-initiate backup, after failure
Collect log
/usr/openv/netbackup/b In/vxlogview-p 51216-o 111-t 00:10:00-d all >/tmp/111.txt
/usr/openv/netbackup/bin/vxlogview-p 51216-o 117- T 00:10:00-d all >/tmp/117.txt
/usr/openv/netbackup/bin/vxlogview-p 51216-o 118-t 00:10:00-d all >/TMP/11 8.txt

/usr/openv/netbackup/bin/admincmd/nbemmcmd-listhost-verbose >/tmp/nbemmcmd.txt
/usr/openv/netbackup/bin/admincmd/bperror-u >/tmp/bperror.txt
/usr/openv/netbackup/bin/admincmd/bppllist-allpolicies-l >/tmp/pol.txt
/usr/openv/netbackup/bin/admincmd/bpstulist-l >/tmp/stu.txt
/usr/openv/netbackup/bin/bpps-x >/tmp/bpps.txt

/usr/openv/volmgr/bin/vmoprcmd-d >/tmp/vmoprcmd_me.txt

/usr/openv/volmgr/bin/tpautoconf-t >/tmp/tpautoconf.txt
/usr/openv/volmgr/bin/tpconfig-d >/tmp/tpconfig.txt
/usr/openv/volmgr/bin/scan >/tmp/scan.txt
/usr/openv/volmgr/bin/vmglob-listall-b >/tmp/vmglob.txt

/usr/openv/netbackup/bin/admincmd/nbrbutil-dump >/tmp/nbrb.txt

/usr/openv/volmgr/bin/vmoprcmd >/tmp/vmoprcmd.txt

/usr/openv/netbackup/bin/admincmd/nbemmcmd-addhost-machinename Media_server_name-machinetype Media- Netbackupversion 7.1-operatingsystem Hpux
/usr/openv/netbackup/bin/admincmd/nbemmcmd-deletehost-machinename Media_server_name-machinetype Media

Nbemmcmd-deletealldevices-machinename Media_server_name-machinetype Media

////////////////////////

Remove the cleaning strips and re-follow the configuration below.
1. Remove the cleaning tape
2. Re-do robot inventory, set the type of cleaning belt, put in the NONE pool, cleaning belt type and tape drive type, such as tape drive is hcart2, the cleaning belt is the "cleaning tape 2 type
3. Double-click the cleaning belt, set the number of cleanings remaining > New account, enter a number, such as 30
4. Check the cleaning belt for the remaining cleaning times, use the command install_path\volmgr\bin\vmquery-m media_id_cleaning_tape to check the output cleanings left parameter

///////////////////////

Nbemmcmd-machinealias-getaliases-machinename Server_name-machinetype Master

Emm Startup failed//////////

1. Confirm the remaining space on the disk where the NBU master server resides. NBU need the remaining space for the total disk space of 10%, such as the C disk 40GB size, need to have more than 4GB of space, otherwise NBU work will be abnormal.
2. Run the following command, and send me the command and output information you executed.
install_path\netbackup\bin\nbdb_ping > C:\nbdb_ping.txt
Install_path\netbackup\bin\bpps > C:\bpps.txt
Install_path\netbackup\bin\ipconfig/all > C:\ip.txt

Install_path\netbackup\bin\admincmd\bpgetconfig > C:\nbuconfig.txt
Install_path\netbackup\bin\admincmd\bpminlicense-list_keys-verbose > C:\nbulic.txt

Install_path\netbackupdb\data\dir > C:\db_dir.txt
Install_path\netbackupdb\log\dir > C:\log_dir.txt

3. Collect the following documents to send to me.
C:\windows\system32\drivers\etc\hosts
Install_path\netbackupdb\log\server.log

Re-configure the Devices////////////////////////////////

1. Cancel the current other backup job to execute the following command,
/usr/openv/netbackup/bin/admincmd/nbrbutil-resetall
/usr/openv/netbackup/bin/admincmd/nbrbutil-dump (This command confirms the output of No resource information)

2. Remove the tape library device on the master server,
/usr/openv/netbackup/bin/admincmd/nbemmcmd-deletealldevices-machinename Szbk52svc-machinetype Media
Check confirmation via command, no device output.
/usr/openv/volmgr/bin/tpconfig-d

3. Stop NBU Service
/usr/openv/netbackup/bin/goodies/netbackup stop
/usr/openv/netbackup/bin/bpps-x (no NB and mm processes are confirmed except pbx_exchange process)
/usr/openv/netbackup/bin/goodies/netbackup start
/usr/openv/netbackup/bin/bpps-x >/tmp/bpps_restart.txt

4. Configure the tape library device
/USR/OPENV/VOLMGR/BIN/TPAUTOCONF-T Check the tape drive, you should be able to list the tape drives of 8 tape libraries
/usr/openv/volmgr/bin/tpautoconf-r Inspection Manipulator
/usr/openv/volmgr/bin/tpautoconf-a Configuring the discovered devices into the NBU

5. Restart the NBU service
/usr/openv/netbackup/bin/goodies/netbackup stop
/usr/openv/netbackup/bin/bpps-x (no NB and mm processes are confirmed except pbx_exchange process)
/usr/openv/netbackup/bin/goodies/netbackup start

6. Check the status of the tape drive
/usr/openv/volmgr/bin/tpconfig-d >/tmp/tpconfig.txt
/usr/openv/volmgr/bin/vmoprcmd >/tmp/vmoprcmd_1.txt
/usr/openv/volmgr/bin/vmoprcmd-d >/tmp/vmoprcmd_2.txt


////////////////////////////////////////////////////////

Troubleshooting Oracle Problems with 1.1.1 NBU backup