Audit vault Server & Agent Installation

Installing the Audit Vault Server

1.         Open a terminal window and “unset” the ORACLE_SID and ORACLE_HOME environment variables. (Note: The environment variables are set for your source database by default.)

            [oracle@EDRSR2P1 ~]$ echo $ORACLE_SID

            orcl

            [oracle@EDRSR2P1 ~]$ export ORACLE_SID=

            [oracle@EDRSR2P1 ~]$ echo $ORACLE_SID

            [oracle@EDRSR2P1 ~]$ echo $ORACLE_HOME

            /u01/app/oracle/product/10.2.0/db_1

            [oracle@EDRSR2P1 ~]$ export ORACLE_HOME=

            [oracle@EDRSR2P1 ~]$ echo $ORACLE_HOME

2.         Install the Audit Vault Server. Invoke Oracle Universal Installer to begin the installation of the Audit Vault server.

a.         The Audit Vault Server installation files have been staged in the $HOME/av_installmedia directory. Change to the directory containing the Audit Vault Server installation files.

            [oracle@EDRSR2P1 ~]$ cd av_installmedia/server 

b.         Invoke the Oracle installer as follows:

            [oracle@EDRSR2P1 server]$ ./runInstaller

3.  Perform an Advanced installation of the Audit Vault Server.

a.         On the Select Installation Type page, select Advanced Installation. Click Next.

b.         On the Advanced Installation Details page, provide the following information and click Next:

            Audit Vault Name: av

            Audit Vault Home: /u01/app/oracle/product/10.2.0/av_1

            Audit Vault Administrator: avadmin1

            Administrator Password: oracle_1

            Audit Vault Auditor Username: avaudit1

            Auditor Password: oracle_1

c.         The Database Vault User Credentials Page appears. Enter the following information:

            Database Vault Owner: dbvowner1

            Owner Password: dbvoracle_1

            Database Vault Account Manager: dbvacct1

            Account Manager Password: dbvoracle_1

            Click Next.

d.         The Product-Specific Prerequisite Checks page appears. After the checks complete, click Next.

e.         The Specify Database Storage Option page appears. Select File System and specify /u01/app/oracle/oradata/av in the Specify Database file location field. Click Next.

f.          The Specify Backup and Recovery Options page appears. Select Enable Automated Backups. Specify /u01/app/oracle/flash_recovery_area/av in the Recovery Area Location field. Specify oracle in the Username and Password fields. Click Next.

g.         The Specify Schema Database Passwords page appears. Select “Use the same passwords for all accounts” and enter oracle_2 in the Password field. Click Next.

h.         Review the information on the Summary page. Click Install.

i.          The Install page appears.

j.          The Configuration Assistants page appears. The Database Configuration Assistant Page appears.

k.         Click OK on the password management page.

l.          Execute the root.sh script as directed. Do not overwrite the existing files.

m.        When the End of Installation page appears, make note of the Enterprise Manager Database Control URL and the Audit Vault Console URL. Click Exit.

            Enterprise Manager Database Control URL: _____________________________

            Audit Vault Console URL: ___________________________________________

n.         Click Yes to confirm.

4. Create a user in the Audit Vault Server database for the Audit Vault Agent.

a.         Open a terminal window and set the Audit Vault environment variables (ORACLE_SID, ORACLE_HOME and PATH) for the Audit Vault Server.

            You can also use the $HOME/labs/setavserver.sh script to accomplish this as follows:

[oracle@EDRSR2P1 labs]$ . ./setavserver.sh

            OR set the environment variables manually:

[oracle@EDRSR2P1 ~]$ export ORACLE_SID=av

[oracle@EDRSR2P1 ~]$ echo $ORACLE_SID

            av

[oracle@EDRSR2P1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/av_1

[oracle@EDRSR2P1 ~]$ echo $ORACLE_HOME

/u01/app/oracle/product/10.2.0/av_1

[oracle@EDRSR2P1 ~]$ export PATH=/u01/app/oracle/product/10.2.0/av_1/bin:$PATH

b. Log in to SQL*Plus using the Database Vault Account Manager username and password.           

[oracle@EDRSR2P1 av_1]$ sqlplus dbvacct1/dbvoracle_1

c.  Create the Audit Vault Agent user. Exit from SQL*Plus.

SQL> create user avagentuser identified by avagentpass;

User created.

SQL> exit

5.  In the terminal window, use AVCA to add the Audit Vault Agent user to Audit Vault. Supply the following values:

            Agentname: avagent1

            Agenthost: <server name> Note: Include the domain name as in this example:       edrsr2p1.us.oracle.com

            Agentusr: avagentuser

[oracle@EDRSR2P1 av_1]$ avca add_agent -agentname avagent1 -agenthost <Audit Vault Agent hostname>  -agentusr avagentuser

AVCA started

Adding agent...

Agent added successfully.

Install the Audit Vault agent on Source Database Server

6.         Invoke Oracle Universal Installer to begin the installation of the Audit Vault agent.

a.         Change directories to the directory containing the Audit Vault agent installation files

 [oracle@EDRSR2P1 ]$ cd av_installmedia/avagent/linux_x32

b.         Invoke Oracle Universal Installer as follows:

[oracle@EDRSR2P1 linux_x32]$ ./runInstaller

7.         Proceed with the installation of the Audit Vault Agent.

a.         On the Audit Vault Agent Installation Details page, provide the following information and click Next:

            Audit Vault Agent Name: avagent1

            Audit Vault Agent Home: /u01/app/oracle/product/10.2.0/av_agent_1

            Agent Username: avagentuser

            Agent Password: avagentpass

            Audit Vault Server Connection Information: host name:listener port:service name

            (auditvault-server-host:1522:av.us.oracle.com as an example)

b.         The Product-Specific Prerequisite Checks page appears. After the checks complete, click Next.

c.         Review the information on the Summary page. Click Install.

d.         The Install page appears.

e.         The Configuration Assistants page appears.

f.          Execute the root.sh script as directed. Click OK.

g.         On the End of Installation page, note the Audit Vault agent port number. Click Exit.

Starting Audit Vault Server & Agent (For Information only)          

f.  Execute the AVCA redeploy command as follows:

            [oracle@EDRSR2P1 av_installmedia]$ avca redeploy

            AVCA started

            Starting OC4J...

            OC4J started successfully.

            Deploying to standalone OC4J...

            done.

            Restarting agent OC4J...

            OC4J restarted successfully.

            [oracle@EDRSR2P1 av_installmedia]$

           

g.  In your Audit Vault Agent terminal window, restart the Audit Vault Agent OC4J as follows:

            [oracle@EDRSR2P1 av_installmedia]$ avctl start_oc4j

            AVCTL started

            Starting OC4J...

            OC4J started successfully.

            [oracle@EDRSR2P1 av_installmedia]$

h.  In your Audit Vault Server terminal window, restart the Audit Vault agent as follows:

            [oracle@EDRSR2P1 av_1]$ avctl start_agent -agentname avagent1

            AVCTL started

            Starting agent...

            Agent started successfully.

11. Log in to Audit Vault Console as the Audit Vault Administrator user.

           

Open a browser window and enter the following URL:

http://<host name>:<Audit Vault Console port>/av

Note: The Audit Vault Console port is displayed at the end of the Audit Vault Server installation. Refer to page A-5 if you have forgotten the port number. Supply the following:

User Name: avadmin1

Password: oracle_1

Connect as: AV_ADMIN

The Oracle Database 10g Licensing Information page appears. Click “I Agree.” Click the Agent tab and verify that the avagent1 agent is started.

Dataguard RMAN

Information in this document applies to any platform.
Oracle Server Enterprise Edition - Version: 10.2.0.1 to 10.2.0.4

Achievement

The following note describes step-by-step procedure to create physical standby by using RMAN duplicate without shutting down the primary (Production) database. Database Name :- prim
Primary db_unique_name :- prim
standby db_unique_name :- stdby

Primary Hostname :- raca.idc.oracle.com
standby Hostname :- core1.idc.oracle.com

Solution

STEP: 1

Enable Force Logging on primary,

SQL> ALTER DATABASE FORCE LOGGING;

Database altered.

 NOTE: Create password file if not present, also check if archiving enabled.

STEP: 2

Configure a Standby Redo Log on primary,

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 3 '/u01/app/oracle/databases/prim/redo/log3a.log' size 50m;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 '/u01/app/oracle/databases/prim/redo/log4a.log' size 50m;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 5 '/u01/app/oracle/databases/prim/redo/log5a.log' size 50m;

Database altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE GROUP 6 '/u01/app/oracle/databases/prim/redo/log6a.log' size 50m;

Database altered.

NOTE: To check the number of SRL,

(maximum number of logfiles for each thread + 1) * maximum number of threads

For example, if the primary database has 3 log files for each thread and 2 threads, then 8 standbys redo log file groups are needed on the standby database.

Verify the standby redo log file groups were created.

SQL> SELECT GROUP#,ThREAD#,SEQUENCE#,ARCHIVED,STATUS FROM V$STANDBY_LOG;

STEP :3

Modify the primary initialization parameter for dataguard on primary,

SQL>alter system set LOG_ARCHIVE_CONFIG='DG_CONFIG=(prim,stdby)';

System altered.

SQL> alter system set LOG_ARCHIVE_DEST_1='LOCATION=/u01/app/oracle/databases/prim/redo/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=prim';

System altered.

SQL>alter system set LOG_ARCHIVE_DEST_2='SERVICE=stdby LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=stdby';

System altered.

SQL> alter system set LOG_ARCHIVE_DEST_STATE_1=ENABLE;

System altered.

SQL>alter system set FAL_SERVER=stdby;

System altered.

SQL>alter system set FAL_CLIENT=prim;

System altered.

SQL>alter system set DB_FILE_NAME_CONVERT='/u01/app/oracle/databases/stdby/data/','/u01/app/oracle/databases/prim/data' scope=spfile;

System altered.

SQL>alter system set LOG_FILE_NAME_CONVERT='/u01/app/oracle/databases/stdby/redo/','/u01/app/oracle/databases/prim/redo' scope=spfile;

System altered.

STEP:4

Run the backup job at the primary by connecting to target and catalog DB(if any)

RMAN> backup as compress backupset database include current controlfile for standby;

On Sql Prompt

SQL> alter system archive log current;

run
{
allocate channel c1 type disk;
allocate channel c2 type disk; backup as compressed backupset database format '/u01/app/oracle/databases/stage/%U';
backup current controlfile for standby format '/u01/app/oracle/databases/stage/%U';

sql ‘alter system archive log current’
backup archivelog all format '/u01/app/oracle/databases/stage/%U';
}


using target database control file instead of recovery catalog
allocated channel: c1
channel c1: sid=22 devtype=DISK

allocated channel: c2
channel c2: sid=32 devtype=DISK

Starting backup at 05-MAR-09
channel c1: starting full datafile backupset
channel c1: specifying datafile(s) in backupset
input datafile fno=00001 name=/u01/app/oracle/databases/prim/data/system01.dbf
input datafile fno=00002 name=/u01/app/oracle/databases/prim/data/undotbs01.dbf
channel c1: starting piece 1 at 05-MAR-09
channel c2: starting full datafile backupset
channel c2: specifying datafile(s) in backupset
input datafile fno=00003 name=/u01/app/oracle/databases/prim/data/sysaux01.dbf
channel c2: starting piece 1 at 05-MAR-09
channel c1: finished piece 1 at 05-MAR-09
piece handle=/u01/app/oracle/databases/stage/0tk95ldn_1_1 tag=TAG20090305T143325 comment=NONE
channel c1: backup set complete, elapsed time: 00:01:56
channel c1: starting full datafile backupset
channel c1: specifying datafile(s) in backupset
including current control file in backupset
channel c1: starting piece 1 at 05-MAR-09
channel c2: finished piece 1 at 05-MAR-09
piece handle=/u01/app/oracle/databases/stage/0uk95ldn_1_1 tag=TAG20090305T143325 comment=NONE
channel c2: backup set complete, elapsed time: 00:01:58
channel c2: starting full datafile backupset
channel c2: specifying datafile(s) in backupset
channel c1: finished piece 1 at 05-MAR-09
piece handle=/u01/app/oracle/databases/stage/0vk95lhb_1_1 tag=TAG20090305T143325 comment=NONE
channel c1: backup set complete, elapsed time: 00:00:03
including current SPFILE in backupset
channel c2: starting piece 1 at 05-MAR-09
channel c2: finished piece 1 at 05-MAR-09
piece handle=/u01/app/oracle/databases/stage/10k95lhd_1_1 tag=TAG20090305T143325 comment=NONE
channel c2: backup set complete, elapsed time: 00:00:02
Finished backup at 05-MAR-09

Starting backup at 05-MAR-09
current log archived
channel c1: starting archive log backupset
channel c1: specifying archive log(s) in backup set
input archive log thread=1 sequence=5 recid=1 stamp=679686819
input archive log thread=1 sequence=6 recid=2 stamp=679686958
input archive log thread=1 sequence=7 recid=3 stamp=679687040
input archive log thread=1 sequence=8 recid=4 stamp=679743155
input archive log thread=1 sequence=9 recid=5 stamp=680047441
input archive log thread=1 sequence=10 recid=6 stamp=680621993
channel c1: starting piece 1 at 05-MAR-09
channel c2: starting archive log backupset
channel c2: specifying archive log(s) in backup set
input archive log thread=1 sequence=11 recid=7 stamp=680636808
input archive log thread=1 sequence=12 recid=22 stamp=680703332
input archive log thread=1 sequence=13 recid=32 stamp=680704306
input archive log thread=1 sequence=14 recid=34 stamp=680704331
input archive log thread=1 sequence=15 recid=36 stamp=680704336
input archive log thread=1 sequence=16 recid=38 stamp=680705571
input archive log thread=1 sequence=17 recid=51 stamp=680711731
channel c2: starting piece 1 at 05-MAR-09
channel c2: finished piece 1 at 05-MAR-09
piece handle=/u01/app/oracle/databases/stage/12k95lhl_1_1 tag=TAG20090305T143531 comment=NONE
channel c2: backup set complete, elapsed time: 00:00:26
channel c1: finished piece 1 at 05-MAR-09
piece handle=/u01/app/oracle/databases/stage/11k95lhl_1_1 tag=TAG20090305T143531 comment=NONE
channel c1: backup set complete, elapsed time: 00:01:01
Finished backup at 05-MAR-09

Starting backup at 05-MAR-09
channel c1: starting full datafile backupset
channel c1: specifying datafile(s) in backupset
including standby control file in backupset
channel c1: starting piece 1 at 05-MAR-09
channel c1: finished piece 1 at 05-MAR-09
piece handle=/u01/app/oracle/databases/stage/13k95lji_1_1 tag=TAG20090305T143634 comment=NONE
channel c1: backup set complete, elapsed time: 00:00:02
Finished backup at 05-MAR-09
released channel: c1
released channel: c2

STEP :5

Create parameter file on primary copy this & password file it to standby and make the necessary changes,

SQL>create pfile='/u01/app/oracle/databases/prim/stage/initstdby.ora' from spfile;

File created.

oracle@raca prim]$ scp –p /u01/app/oracle/dbs/initstdby.ora standbyhost:/u01/app/oracle/dbs/

oracle@raca prim]$ scp –p /u01/app/oracle/dbs/Pwdorcl standbyhost:/u01/app/oracle/dbs/

STEP :6

Copy the listener.ora,tnsname.ora and sqlnet.ora files into staging directory.

[oracle@raca prim]$ scp –p /u01/app/oracle/network/admin/*.ora standbyhost:/u01/app/oracle/network/admin

Copy the redo logs also to standby to avoid size mismatch,

[oracle@raca prim]$ scp –p /u01/app/oracle/databases/prim/redo/*.log standbyhost:/u01/app/oracle/databases/prim/redo/

STEP : 7

Move the files to standby server,

a. Create the same directory on standby server and copy the backups. os standby,

$mkdir -p /u01/app/oracle/databases/stage/

[oracle@raca prim]$ scp /u01/app/oracle/databases/stage/* core1.idc.oracle.com:/u01/app/oracle/databases/stage/

b. Alternatively we can copy the backups to standby different directory. On standby server connect to RMAN target as primary and auxiliary instance or if your primary is having catalog database then connect to target as primary, catalog database and auxiliary instance. Catalog those backup pieces to let the controlfile of primary or catalog database to know the backup information.

For details on how to catalog backup piece refer,

NOTE 470463.1 - How To Catalog Backups / Archivelogs / Datafile Copies / Controlfile Copies

STEP: 8

Make the necessary changes on the copied initstdby.ora file on standby.

db_name=prim
db_unique_name=stdby
log_archive_config='DG_CONFIG=(prim,stdby)'
log_archive_dest_1='LOCATION=/u01/app/oracle/databases/stdby/redo/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=stdby'
log_archive_dest_2='SERVICE=prim LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=prim'
log_archive_dest_state_1=ENABLE
db_file_name_convert='/u01/app/oracle/databases/prim/data/','/u01/app/oracle/databases/stdby/data/'
log_file_name_convert='/u01/app/oracle/databases/prim/redo/','/u01/app/oracle/databases/stdby/redo/'
REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE
standby_archive_dest='/u01/app/oracle/databases/stdby/arch/'
standby_file_management=AUTO
fal_client=stdby
fal_server=prim

STEP: 9
Create standby instance,
create the same password as that of primary. Alternatively we can copy the password file from primary and rename it on standby.
For example from primary copry the password file,
scp /u01/app/oracle/dbs/opapwprim core1.idc.oracle.com:/u01/app/oracle/dbs/
on standby,
$mv orapwprim orapwstdby
or create new password same as primary as mentioned below,

a. in UNIX,

$export ORACLE_SID
$orapwd file='$ORACLE_HOME/dbs/orapwstdby' password=sys entries=10;

 in Windows,

d:>set ORACLE_SID=stdby
d:>oradim -new -sid stdby -intpwd sys

b Create necessary directories and start the auxiliary instance on standby 

[oracle@core1 stdby]$ pwd
/u01/app/oracle/databases/stdby

[oracle@core1 stdby]$mkdir data redo oradata stage udump bdump cdump

SQL>create spfile from pfile='/u01/app/oracle/databases/stage/initstdby.ora';

File created.

SQL>start nomount;

ORACLE instance started.

Total System Global Area 612368384 bytes
Fixed Size 1220844 bytes
Variable Size 167776020 bytes
Database Buffers 436207616 bytes
Redo Buffers 7163904 bytes


SQL>exit

STEP :10

Create net services on both primary and standby,

At prim server,
prim =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =raca.idc.oracle.com )(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = prim)
)
)

stdby =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =core1.idc.oracle.com )(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = stdby)
)
)


At standby server,
prim =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =raca.idc.oracle.com )(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = prim)
)
)

stdby =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =core1.idc.oracle.com )(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = stdby)
)
)

STEP : 11  Use RMAN duplicate for standby on standby,
When 'dorecover' is specified in the duplicate for stanby command then do a archivelog switch on the target database and run the duplicate.

Example : on primary,
sql> alter system archive log current;
on standby,

Connect to target(i.e. prim via connect string) and catalog database(if any) and this auxiliary database by,

$RMAN target sys/oracle@prim catalog RMAN/RMAN@catdb auxiliary / if no catalog database,

$RMAN target sys/<password>@prim auxiliary /

Recovery Manager: Release 10.2.0.1.0 - Production on Thu Mar 5 12:30:56 2009

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database: PRIM (DBID=3971986030)
connected to auxiliary database: PRIM (DBID=3971986030, not open)

RMAN> duplicate target database for standby dorecover;

Starting Duplicate Db at 05-MAR-09
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=36 devtype=DISK

contents of Memory Script:
{
set until scn 503194;
restore clone standby controlfile;
sql clone 'alter database mount standby database';
}
executing Memory Script

executing command: SET until clause

Starting restore at 05-MAR-09
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/databases/prim/stage/0pk86cs2_1_1
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/databases/prim/stage/0pk86cs2_1_1 tag=TAG20090224T175722
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
output filename=/u01/app/oracle/databases/stdby/oradata/control01.ctl
Finished restore at 05-MAR-09

sql statement: alter database mount standby database
released channel: ORA_AUX_DISK_1

contents of Memory Script:
{
set until scn 503194;
set newname for datafile 1 to
"/u01/app/oracle/databases/stdby/data/system01.dbf";
set newname for datafile 2 to
"/u01/app/oracle/databases/stdby/data/undotbs01.dbf";
set newname for datafile 3 to
"/u01/app/oracle/databases/stdby/data/sysaux01.dbf";
restore
check readonly
clone database
;
}
executing Memory Script

executing command: SET until clause

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 05-MAR-09
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: sid=36 devtype=DISK

channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00003 to /u01/app/oracle/databases/stdby/data/sysaux01.dbf
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/databases/stage/0uk95ldn_1_1
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/databases/stage/0uk95ldn_1_1 tag=TAG20090305T143325
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:07
channel ORA_AUX_DISK_1: starting datafile backupset restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /u01/app/oracle/databases/stdby/data/system01.dbf
restoring datafile 00002 to /u01/app/oracle/databases/stdby/data/undotbs01.dbf
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/databases/stage/0tk95ldn_1_1
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/databases/stage/0tk95ldn_1_1 tag=TAG20090305T143325
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:15
Finished restore at 05-MAR-09

contents of Memory Script:
{
switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy recid=4 stamp=680711065 filename=/u01/app/oracle/databases/stdby/data/system01.dbf
datafile 2 switched to datafile copy
input datafile copy recid=5 stamp=680711065 filename=/u01/app/oracle/databases/stdby/data/undotbs01.dbf
datafile 3 switched to datafile copy
input datafile copy recid=6 stamp=680711065 filename=/u01/app/oracle/databases/stdby/data/sysaux01.dbf

contents of Memory Script:
{
set until scn 503194;
recover
standby
clone database
delete archivelog
;
}
executing Memory Script

executing command: SET until clause

Starting recover at 05-MAR-09
using channel ORA_AUX_DISK_1

starting media recovery

channel ORA_AUX_DISK_1: starting archive log restore to default destination
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=11
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=12
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=13
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=14
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=15
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=16
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=17
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/databases/stage/12k95lhl_1_1
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/databases/stage/12k95lhl_1_1 tag=TAG20090305T143531
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_AUX_DISK_1: starting archive log restore to default destination
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=8
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=9
channel ORA_AUX_DISK_1: restoring archive log
archive log thread=1 sequence=10
channel ORA_AUX_DISK_1: reading from backup piece /u01/app/oracle/databases/stage/11k95lhl_1_1
channel ORA_AUX_DISK_1: restored backup piece 1
piece handle=/u01/app/oracle/databases/stage/11k95lhl_1_1 tag=TAG20090305T143531
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:15
archive log filename=/u01/app/oracle/databases/stdby/redo/1_8_679684398.dbf thread=1 sequence=8
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_8_679684398.dbf recid=8 stamp=680711072
archive log filename=/u01/app/oracle/databases/stdby/redo/1_9_679684398.dbf thread=1 sequence=9
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_9_679684398.dbf recid=10 stamp=680711078
archive log filename=/u01/app/oracle/databases/stdby/redo/1_10_679684398.dbf thread=1 sequence=10
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_10_679684398.dbf recid=9 stamp=680711078
archive log filename=/u01/app/oracle/databases/stdby/redo/1_11_679684398.dbf thread=1 sequence=11
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_11_679684398.dbf recid=6 stamp=680711066
archive log filename=/u01/app/oracle/databases/stdby/redo/1_12_679684398.dbf thread=1 sequence=12
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_12_679684398.dbf recid=7 stamp=680711067
archive log filename=/u01/app/oracle/databases/stdby/redo/1_13_679684398.dbf thread=1 sequence=13
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_13_679684398.dbf recid=4 stamp=680711066
archive log filename=/u01/app/oracle/databases/stdby/redo/1_14_679684398.dbf thread=1 sequence=14
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_14_679684398.dbf recid=1 stamp=680711066
archive log filename=/u01/app/oracle/databases/stdby/redo/1_15_679684398.dbf thread=1 sequence=15
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_15_679684398.dbf recid=3 stamp=680711066
archive log filename=/u01/app/oracle/databases/stdby/redo/1_16_679684398.dbf thread=1 sequence=16
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_16_679684398.dbf recid=2 stamp=680711066
archive log filename=/u01/app/oracle/databases/stdby/redo/1_17_679684398.dbf thread=1 sequence=17
channel clone_default: deleting archive log(s)
archive log filename=/u01/app/oracle/databases/stdby/redo/1_17_679684398.dbf recid=5 stamp=680711066
media recovery complete, elapsed time: 00:00:01
Finished recover at 05-MAR-09
Finished Duplicate Db at 05-MAR-09

RMAN>exit

NOTE: We can use without dorecover clause also,

Example,

 RMAN>duplicate target database for standby;

OR, this is for when standby database have same directory structure

 RMAN>duplicate target database for standby nofilenamecheck dorecover;

STEP : 12 Start the MRP(managed recovery process) on standby,

SQL> select name,db_unique_name,database_role, from v$database;

NAME DB_UNIQUE_NAME DATABASE_ROLE
--------- ---------------- --------------
PRIM STDBY PHYSICAL STANDBY


SQL> alter database recover managed standby database disconnect;

Database altered.

STEP : 13

Enable the log_archive_dest_2 on primary, which is to send the logs to standby server.

SQL>alter system set log_archive_dest_state_2=enable;

System altered.

STEP :14

Check the standby whether it is in SYNC with primary,

A. Check the v$archived view on standby,

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# FIRST_TIM NEXT_TIME
---------- --------- ---------
29 06-MAR-09 06-MAR-09
30 06-MAR-09 06-MAR-09
31 06-MAR-09 06-MAR-09
32 06-MAR-09 06-MAR-09
33 06-MAR-09 06-MAR-09
34 06-MAR-09 06-MAR-09
35 06-MAR-09 06-MAR-09
36 06-MAR-09 06-MAR-09
37 06-MAR-09 06-MAR-09
38 06-MAR-09 06-MAR-09
39 06-MAR-09 06-MAR-09

B. Do the log switch on primary,

SQL> ALTER SYSTEM SWITCH LOGFILE;

System altered.

C. On standby,

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;


SEQUENCE# FIRST_TIM NEXT_TIME
---------- --------- ---------
29 06-MAR-09 06-MAR-09
30 06-MAR-09 06-MAR-09
31 06-MAR-09 06-MAR-09
32 06-MAR-09 06-MAR-09
33 06-MAR-09 06-MAR-09
34 06-MAR-09 06-MAR-09
35 06-MAR-09 06-MAR-09
36 06-MAR-09 06-MAR-09
37 06-MAR-09 06-MAR-09
38 06-MAR-09 06-MAR-09
39 06-MAR-09 06-MAR-09

SEQUENCE# FIRST_TIM NEXT_TIME
---------- --------- ---------
40 06-MAR-09 07-MAR-09


SQL> select sequence#,applied from v$archived_log order by sequence#;


SEQUENCE# APP
---------- ---
29 YES
30 YES
31 YES
32 YES
33 YES
34 YES
35 YES
36 YES
37 YES
38 YES
39 YES

SEQUENCE# APP
---------- ---
40 YES

Avadon: The Black Fortress Released.

On Monday, after fifteen months of work, Spiderweb Software released Avadon: The Black Fortress.

I'm writing this on Wednesday, after a week spent in a pretty continuous state of total freaking out. I am always nervous when releasing a new game. But a new game in a new world with a new system? After spending an unusual amount of time on it? I've been going kind of insane.

For those who are interested, early sales are quite strong. I'm starting to suspect that we aren't about to go out of business. And yet, most of the early sales are to die-heard fans. The question is how many new people will play it and like it. I think that it's solid, the world is cool, and the game itself is a lot of fun to play. But I might be wrong. It happens all the time.

Releasing a new game also means that I have to read my forums, which, as I have written before, is painful. Even for a solid game, most of the things that are written will be critical. It's only with time that I can get a read on how good the good parts are.

This is also when I have to take the lumps for unpopular but necessary choices I had to make. The biggest complaint is that the game has no keyboard movement. Our other big series, Avernum, takes place on a fairly simple grid, so keyboard movement is easy to implement. Avadon takes place in a larger world with no simple grid for the characters to stand on, and keyboard movement just doesn't work as well as the mouse, especially for distances that aren't very short. But some people really want keyboard movement, and I can hardly tell them they are wrong. I just have to take the criticism and hope that the game is good enough.

I've also been criticized that the game text is too small, and I'm taking that to heart. I am going to work on using a larger font for dialogue and special encounters, which should help a lot.

But now I'm rambling. Releasing a game is only the beginning of a long process. Maintenance. PR. Sequels. I hope you try it out. I hope you like it. And I hope Spiderweb gets to stay in business.

On to the Windows port (hopefully out at the end of April). And the iPad port (I have no idea when this might be ready).

Three Rules For Difficulty In RPGs

As I put the final touches on Avadon: The Black Fortress, I am finally wrapping up the most touchy, painful, contentious part of the game: play balance. Should it be harder or easier? Is this particular fight fair? How many players will wash out halfway through the game and be forever angry at me for my suckitude?

The fans of role-playing games are a pretty diverse bunch. Some prefer stories. Some fixate on stat-building. Some want to beat everything easily, and others are irritated if there are no challenges. It's a pretty amorphous blob of interests, and nobody can satisfy them all. I just try to appeal to as much of the blob as possible.

Over the years, I have made three observations regarding game difficulty. These are the ideas I always come back to as I decide how tough a given section or encounter will be. I think these guidelines are, to the extent anything can be in a creative endeavor, The Truth.

Observation One: There are two sorts of fights in an RPG: Fights that are supposed to be easy and fights that are supposed to provide a challenge.

In other words, first, there are fights that will almost never ever kill a player, also known as trash, or trash mobs. If your trash mobs are frequently killing the character, your balance is messed up. (Early versions of Avernum 6 had a big problem with this.) Most of the time, the vast majority of the fights in a game will be this sort.

Then there are fights that the player can possibly lose (mini bosses, bosses). And, of course, for fights that can kill the player, there is a spectrum of how likely that end result is. Some bosses will only kill you if you really aren't paying attention. Others require actual skill and strategy, and maybe a few tries to get your tactics down.

Observation Two: In an RPG, you have to have some of both sorts of fight.

RPG fans expect a lot of trash to slaughter, so they can be a badass and Conan-like and so they can collect experience to get strong and get new spells and swords and stuff.

The trickier part is understanding the need for tough fights. Very often, players don't like to be seriously challenged. They hate to lose. They hate to lose repeatedly. Sometimes, the temptation to just give up and have every fight on the default difficulty be easy peasy is overwhelming.

But you still need to have tough fights, for several reasons. A game full of only easy fights against trash is monotonous and dull. The suspension of disbelief in a role-playing game is delicate, and, if a dragon is only as tough as Goblin #0145, it just feels wrong. And because the adrenaline rush of achieving something difficult (be it slaying a demon lord of winning a game of solitaire) is one of the great pleasures of computer games, and you just can't lose that.

So, you might ask, why don't I just put in tough fights and really carefully balance them so that everyone can beat them in just a few tries? That brings us to the third observation, which is both subtle and vitally important.

Observation Three: If a fight has any chance of beating the player, there is a percentage of users who will NEVER be able to beat it.

It took me a long, long time to realize this. Too long. But it is vitally important to understand the difficulty of doing game balance.

Everyone has bad days. Everyone has blind spots. Some people who reach your tough fight will have used up all their healing potions, or refuse to use the healing potions, or forget that they have healing potions, or never have realized that healing potions are potions you can drink that heal you. Because of this, whenever people reach a tough fight, there will be a few of them who just can't beat it. They just can't. You can adjust the percentage of people who lose, but it will never be zero.

(By the way, along these lines, if you put any puzzle or riddle in a game, there is a percentage of users who will never figure it out. This is why I've drastically reduced the number of puzzles in my games. Arguably, I have reduced it too much. It bears thinking about. However, this is my rationale for doing it.)

When someone loses to a fight more than three or four times, they will almost always be angry, and they will always blame you. Some of them will temporarily lower the difficulty, get past the fight, and move on. Some will gut it out and prevail. And some will ragequit and you will never sell a game to them again.

I try to appeal to a wide group of customers, but there is one customer I can never appeal to: The gamer who can't beat a fight and refuses to lower the difficulty. I get e-mails from this person all the time, expressions of hurt and betrayal and rage, accompanied with the promise that they will never buy another one of my games. A promise I believe, by the way. I hate getting these e-mails. Everyone does. It's like a punch in the stomach. But I suck it up, send a nice message back, and move on. It's like death and taxes. It's part of the biz.

But, with every difficult encounter, I eventually have to plant my flag and say, "You must be at least this badass to go on this ride." If you aren't sure why I do this, please consult with the reasons accompanying Observation Two.

Of Course, These Aren't Universal Laws

I expect that the standard swarms of hair-trigger internet nitpickers haven't read this far. They went to the Comments to excoriate me based on some minor point about Observations One or Two, or they want me to explain how Shadow of the Colossus can be a good game with only boss fights. (Answer: Not an RPG.)

But role-playing games are a genre. Genres have conventions. That's what makes them Genres! I can go along with the conventions or fight against them, but, either way, they are part of the DNA of the games I make.

Role-playing games need trash and they need bosses. Put both in, and never lose sight of what makes trash trash and what makes bosses bosses.

Shameless Self-Promotion.

The gaming site GameBanshee was kind enough to call our newest game, Avernum 6, in their annual awards. We were runner-ups in the Independent RPG of the Year and RPG Game of the Year categories. Thank you very much!

This sort of thing is exciting to me, and not just because of the obvious self-esteem and greed motivations. The recession has been really rough on the gaming press, and, because of that, it's been rough on us. Many of the writers and editors who liked to cover us lost their jobs, and many of the small news outlets that were willing to cover humble Indie games like ours disappeared or were absorbed. It's been depressing.

The hardest part about being a small Indie is getting attention. Press coverage is oxygen. It takes the tenacity of the cockroach to eventually find editors who are willing to say, "Sure, they're small, but I believe in what they're doing. I'm going to back them up." And, when it happens (such as the recent coverage of us in PC Gamer), I'm incredibly grateful.

Avadon Developer Diary #5 - Getting It Done.

This will probably be the last developer diary for Avadon: The Black Fortress. We hope to release it for the Macintosh at the end of February. Then the process of porting it to Windows and the iPad shall commence. Of course, I could get the flu or be hit by a truck, which will alter the release schedule slightly.

The first preview of Avadon recently appeared on Inside Mac Games. There's a lot of good information there.

I've already written diaries about the origin of the game, the tone, the sorts of decisions that need to be made, and the character classes. For me, the longest, most grueling part of writing the game is making the scenario. Dungeons have to be designed. Dialogue needs to be written. Gold coins, clay pots, and spoons need to be placed. In other words, the game itself needs to be made.

That process finally ended last week, meaning that we're in the endgame. Time to wrap up all the odd details and kick the game out the door! This is what I generally call the OHGODOHGODITBURNSGETITOFFMEGETITOFFMEEEEEEEEE phase.

This process, during which everyone involved is pretty much running off fumes, has several parts: Odds and Ends, Endgame Balance, Bug Fixing, and Getting the Game Stable.

Odds and Ends

There are always lots of aggravating parts of the job of writing a game that I put off and put off. Now they can't be put off anymore. Every job I hate. Doing the sound effects. And writing the documentation (and hint book). And making the game icon, and the installer. Nothing can be delayed anymore.

A lot of this work is very important, as they determine a player's first impression. Things like the icon and the starting music and art are the first things a player will experience. These details require a lot of attention, so I work on them at a time when I can focus on them as much as possible.

For Avadon, I've put a lot of work into directing the art at the beginning of the game so that they help the player understand a complicated world, with a lot of intricate politics. The early moments of the game can't be wasted. I want to intrigue the player with the story at every possible moment.

Endgame Balance

Balancing the end of the game is particularly difficult. There are several reasons for that. First, there is a wide spread in power levels among players. The dedicated grinders and min-maxers have made characters that are about as powerful as the game allows. More casual players have characters who have fallen farther and farther behind. An encounter that is challenging to a hardcore player will be impossible for a casual player, and an encounter that is tough for a casual player is probably painfully easy for the serious player.

It's a very difficult target to hit, complicated by the fact that at least some of the epic battles at the end of the game should have some pushback to them. They should feel a little tough.

So I try to make the late game fights hard but not impossible for the more casual players. I also try to put in some optional endgame fights that are a real challenge for the toughest players. These two targets are actually tricky to hit precisely, and it requires a lot of feedback and rebalancing.

In Avadon, I want the player to be able to challenge Redbeard for control of the Black Fortress. I want this fight to be soul-crushing, but, with great skill, preparation, and luck, winnable. This is an extra-difficult target to hit. Making something almost impossible is hard. Still, I think I'm getting close.

Bug Fixing and Getting the Game Stable

Of course, this should happen. Now that the game is mostly done, I am getting to the point where everything has been tested and is basically playable. Now that I am trying to actually release the game, I am trying to get it to the most stable, functional point I possibly can.

It's tricky, because Avadon has a lot of player decisions that can make big changes in the ending. Fortunately, my industrious beta testers are doing a great job of trying out all the different possibilities.

This is actually a much more complicated process than it sounds. You see, whenever I make a change, even the most seemingly tiny, innocuous change, there is a chance that I just broke everything.

Games Are Like Giant Cubes of Jello

One of my favorite books on software development describes an unshipped piece of software as a ten by ten by ten foot cube of jello. When you finish it, it is wobbling and shaking. Then, slowly, the vibrations stop and it becomes stable. However, whenever you poke the jello, it starts to wobble again and it takes a long time to become still.

Avadon is a huge cube of jello that is wobbling like mad. As testers play it and don't find serious problems, it stops wobbling. When I make a change, any change at all, I poke it. When the jello is almost still, I go, "OK, I will release the game ... NOW!" and hope it isn't broken. This is how the process works at its best.

So fixing bugs now is a process of triage. When I get a bug report, I think, "Is this serious enough to risk fixing it, bearing in mind that my fix might completely mess up the game?" As we get closer and closer to the ship date, more and more minor issues get kicked off to the v1.0.1 release.

If you've ever wondered why games ship with bugs, this is part of the answer. There is no excuse for releasing a broken game. However, small flaws are always tolerated in order to avoid disaster. Perfection is for v1.0.4.

So Back To Work

If you've been reading these diaries, thank you! I hope I made Avadon sound interesting, and I hope the game is to your liking. It's been a very long road. I've put a lot of myself into this game, and it really is the sort of game I would want to play. Thus, if it turns out to be a failure, I'll have a lot of thinking to do.

Thanks for reading, and I'll see you on the flip side!

Three Reasons Creators Should Never Read Their Forums

If I have learned anything from writing Indie games for a living for fifteen years (and there are plenty who would say that I haven't), it is that it is usually a bad idea for creators to visit online forums discussing them and their work. It doesn't lead to happy ends.

This is why big, smart companies with actual budgets hire community people who do nothing but deal with and sift through forums. Managing fans is real work, and picking out the realistic and worthwhile comments takes a ton of time and judgment. That is why smart companies put a layer between the fans and the creators. If you don't have this layer, you should keep a safe, respectful distance.

It's a pity. My company, Spiderweb Software, has a really awesome, active online forum. Been there for years. Always active, full of all sorts of discussions. However, unless I've just released a game and are looking for signs of early, evil bugs, I have to stay away from it.

Some of my fans really resent this and take it personally, and they haven't been shy about letting me know. But if you've ever wondered why the creators of your beloved games often avoid the forums (especially the Word of Warcraft forums, Yeesh!), this might help you to understand why.

1. It's Not Productive To Read How Much People Hate You

It's been said that, if you want a healthy marriage, you have to say five kind things for every unkind thing. It is in our nature to gloss over and ignore kind words, but to really fixate on and get affected by unkind ones. This is why Facebook will never have a Don't Like button. If you see "Joe likes your post," well, fine. If the average online denizen see "Joe doesn't like this," he or she will probably freak out.

Which brings us to forums.

A few years ago, I wrote an article for IGN about how I felt that Indie games were far from the only source of innovation, and the big companies don't get enough credit for trying to make innovative things. Slashdot was kind enough to link to it. Someone might agree or disagree with me. Fine. But someone wrote this ...

"This is the kind of commentary I'd expect out of a cynical independent ripoff artist in action, really. You know, the kind of person who is too afraid and closed-minded to try anything new, partly because he doesn't want to lose his money or reputation - a sound judgement - and partly because he just doesn't seem to want to try. ...  In other words, nothing to see here. Just near-mindless droning from another cynic with a rather skewed and defeated view of the gaming world."

WHOA! DUDE! What did I ever do to you? Did I run over your dog? Make out with your mom? Go to where you work and mess up the settings on the fry vat? Damn!

Now, I thought that one was pretty funny. I sent links to it to my friends, saying, "Hey! Look what people REALLY think of me!" Over the years, I've developed a pretty thick skin. And yet, if you read lots of people dumping on you, unless you have super-human emotional control, it's eventually going to get to you. Sometimes I'll get weak and look at a forum and see some nasty cheap shot and it'll throw me off my game for hours.

Remember, as Penny Arcade put it (in a far superior and NSFW way), anonymity plus audience makes assholes. (And, for what it's worth, the creators of Penny Arcade don't read their forums either.)

It's a little different on my company's forums. But only a little. Even though it is mainly populated by my fans, it is still full of shots at my design skill, game quality, virility, and facial complexion. Remember, there's a thin line between love and hate. Nobody will lash out at you like a disappointed fan.

When I read the forums for, say, World of Warcraft or xkcd, I'm always amazed at how nasty things get. It makes me think, "If you hate it so much, why are you there?" But that's just the way it is, and excess exposure to insults can really get under your skin, make you doubt yourself, and interfere with your work. It's very sad, but you sometimes need to just protect yourself by staying away. Keep your brain clean.

2. It's Not Going To Be Helpful

It seems like reading forums would be a good way to get design ideas and learn ways to improve your games. With the exception of learning about bugs, this is usually not the case.

It can be tempting, when you're stuck designing a game, to read forums and look for feedback. The problem is this. No matter what the question is, there are people who will advocate strongly for both sides of it. Many of these people reflexively hate change. Many of these people are only happy if the game is much harder (or much easier). Some of them will not, in fact, have a realistic idea about anything. Often, there are issues where intelligent people can come to opposite conclusions, and you can read thousands of furious posts on either side of the issue without getting an inch closer to an actual decision.

Forums contain a cacophony of people telling you to do diametrically opposite things, very loudly, often for bad reasons. There will be plenty of good ideas, but picking them out from the bad ones is unreliable and a lot of work. If you try to make too many people happy at once, you will drive yourself mad. You have to be very, very careful who you let into your head.

Of course, it is still very important for designers to get lots of good, constructive criticism. That is why I have built up an elite cadre of awesome beta testers and interested friends, and I listen to them very closely. And, I must point out, many of those testers were recruited from my forums. You just need to choose carefully the people you ask for advice.

3. You Might Get Suckered Into Getting Angry

Not much to say about this. If you read forums for long enough, you will read a lot of nasty comments and cheap shots. If you read enough cheap shots, you'll get angry. If you are angry enough, you will eventually lash out and flame back.

Snapping angrily at your customers never, ever leads to good results.

One Final Comment

I'm sure some of my fellow Indie developers are reading this and shaking their heads at my idiocy. A lot of developers do maintain close relationships with their forums. It works for them, for now. I'm glad for them, and I hope it keeps working.

Just bear this in mind. When you start out and gain your first following, you get a grace period. You're a fresh face, making awesome new things. Everyone loves you. And, most importantly, you haven't had a chance to start disappointing chunks of your fan base yet.

The longer you are active, the more of your fans will turn on you, justified or not.

And, if you are a member of my forums reading this, know this. I love you guys. The idea that anyone wants to discuss my work at all, even to dump on it, is insanely flattering. I just hope that this makes clearer the instincts of efficiency and self-preservation that lead me to keep a little bit of distance.

Edit: Changed "think skin" to "thick skin."