Get ASM Files Using LFTP

If a database has XML DB set and ftp port configured then lftp could be used to get files out of the ASM.
FTP port could be configured with

SQL> @$ORACLE_HOME/rdbms/admin/catxdbdbca.sql 7001 8001
7001 is the ftp port 
8001 is the http port

Listener status would display these ports

lsnrctl status
...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hostname)(PORT=7001))(Presentation=FTP)(Session=RAW))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hostname)(PORT=8001))(Presentation=HTTP)(Session=RAW))
...

To get a file for example an archive log file use

lftp rac4 -usystem,passwd -p7001 -e 'get /sys/asm/FLASH/RAC11G2/ARCHIVELOG/2011_04_02/thread_2_seq_472.390.747360023;exit;'
41125376 bytes transferred in 4 seconds (11.02M/s)

Here rac4 is the host on which the ASM instance is running, command could be executed from a remote node as well. Following -u is the username used to login into the system, in this case database's system user and comma separated by the password for system user. FTP port is specified with -p.

Without the exit at the end, after the file is transferred shell prompt will end up in the ftp prompt.

To remove the http and ftp ports get rid of the dispatcher parameter (as per 274508.1 Listener Issue: Removing XDB Handlers for HTTP and FTP Ports) or

conn / as sysdba
exec dbms_xdb.sethttpport(0);
exec dbms_xdb.setftpport(0);
alter system register;

More on Master Note for Oracle XML DB Protocols: FTP HTTP HTTPS WebDAV, APEX and Native Database Web Services [ID 1083991.1]

Apart from lftp ftp client such filezilla could be used to connect to the ftp port specified and get the files.

Enable Flashback On Standby

It is possible to enable flashback on a standby independent of the primary. (It's good to have flashback enabled both on primary and standby, if flashback IO is no concern).
Data guard environment created earlier is used here. Steps are similar to How to Enable Flashback for a RAC Database on ASM ( 819905.1)

To enable flashback on standby bring the stop log apply on standby (transport could also be stopped for the duration).

DGMGRL>  edit database rac11g2 set state='TRANSPORT-OFF';
Succeeded.
DGMGRL> edit database rac11g2s set state='APPLY-OFF';
Succeeded.

Trying to enable flashback while log apply is on will result in

SQL> alter database flashback on;
alter database flashback on
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active

If the standby is open in read only mode then shutdown and bring it up only one instance in mount mode as this is required to enable flashback.

srvctl stop database -d rac11g2s
srvctl start instance -d rac11g2s -i rac11g2s1 -o mount

Once in mount mode enable flashback on the standby

SQL> alter database flashback on;

Database altered.

Enable log apply and transport on

DGMGRL> edit database rac11g2 set state='TRANSPORT-ON';
Succeeded.
DGMGRL> edit database rac11g2s set state='APPLY-ON';
Succeeded.

Open the standby mode in read only if required. Check the flashback on status of the database

SQL> select flashback_on from v$database;

FLASHBACK_ON
------------
YES

Having flashback on standby is important especially to recover from open resetlogs situations in primary. If the standby has applied changes past the new resetlog scn and there's no flashback enabled on standby, only way to recover is to recreate the standby. If flashback was enabled on standby then it could be used to flashback to a scn prior to resetlogs and continue to use the standby from then onwards.

Stuck Archiver Processes and FAL Gap Resolution Not Working

On a RAC to RAC data guard configuration (11.1.0.7) network wait class and waits on LGWR LNS seem to appear out of the blue. System has been running for a quite a number of years and there has not been any changes to network or any other hardware (NIC, cables and etc). Output from the emconsole
Drilling down to "other" wait class could see following LGWR LNS wait
The wait histogram showed a constant value of 16ms
Following was observed in the network wait class drill down
Metalink note Data Guard Wait Events (233491.1) describes these wait event as "ARCH wait on ATTACH - This wait event monitors the amount of time spent by all archive processes to spawn an RFS connection. The LGWR-LNS wait on channel wait event is for standby destinations configured with either the LGWR ASYNC or LGWR SYNC=PARALLEL attributes.
LGWR-LNS wait on channel - This wait event monitors the amount of time spent by the log writer (LGWR) process or the network server processes waiting to receive messages on KSR channels."

During this time there was a considerable archive gap between primary and standby and FAL gap resolution seems unable to resolve it. (FAL use to work fine).

The thought of making changes to transport related values in Oracle net (send and receive buffer) was suppressed since there has not been any hardware changes.
Metalink didn't give any more than definition for these waits.
Googling yield a forum post with a mention of metalink note Bug 5576816 - FAL gap resolution does not work with max_connection set in some scenario. This was applicable for 10.2.0.3 not 11.1.0.7 but the recommendation on the posting (found through googling) was to kill all the archive processes in primary(these will get restated as soon as they get killed). Reason given was that these archive processes were "stuck" and need restart. Looking at the emconsole it was also visible waits were happening on the archive processes.

Tried to kill the archive processors "proper way" by changing the log_archive_max_processes to 1 but this didn't kill any of the processes. Even after setting it to 1 all the archive processes were running. Then did a rolling shutdown and start up of the primary which resolved the issue.

Unfortunately the issue was back after few days on one of the nodes. This time killing the Oracle database session of the archive processes waiting for these wait events resolved it. It seem the archive processes being "stuck" is the symptom and cause could be something else.
Blog post will be updated ...