Table of Contents
Problem Statement
One thing I have learned over the years is troubleshooting skills are often more important than technology specialization.
Oracle Middleware is not my primary area of expertise. However, when faced with a persistent issue, a structured troubleshooting approach and a never-give-up attitude can often help uncover the root cause, regardless of the technology stack.
This article documents one such troubleshooting journey involving Oracle Forms and Reports.
After successfully installing Oracle Forms and Reports 12.2.1.4, Forms was working normally but Reports Server refused to start.
The primary error observed was:
REP-52266: The In-Process Reports Server Failed to Start
Additional errors observed during the investigation included:
REP-65003: NetworkUtility:getIOR The request timed out REP-50504: rep_server1 not found in the network REP-56105: Engine rwEng-0 died with error
At first glance, this appeared to be a Reports discovery or communication issue. However, as we started peeling the layers one by one, the actual root cause turned out to be somewhere completely different.
Understanding the Startup Flow
Before jumping into troubleshooting, it is important to understand what happens internally when Oracle Reports starts.
In a typical Oracle Forms and Reports deployment, the Reports Servlet runs within the WebLogic managed server and handles all incoming Reports requests:
WLS_REPORTS
When a user accesses:
http://<host>:<port>/reports/rwservlet/showjobs
The servlet attempts to communicate with the in-process Reports Server.
A simplified startup flow looks like:
Reports Server Startup => Configuration Loading => Security Initialization (OPSS/JPS) => Network Initialization => Reports Engine Startup (rwEng-0) => Reports Server Registration
Since the visible error was:
REP-52266: The In-Process Reports Server Failed to Start
Our first objective was to determine exactly which stage of this startup sequence was failing.
Reviewing Oracle Logs
The first step was to review the Oracle Reports and WebLogic logs before moving into operating system level diagnostics.
The primary files examined were:
$DOMAIN_HOME/servers/WLS_REPORTS/logs/reports/rwservlet_diagnostic.log $DOMAIN_HOME/servers/WLS_REPORTS/logs/reports/rwserver_diagnostic.log $DOMAIN_HOME/servers/WLS_REPORTS/logs/WLS_REPORTS.out $DOMAIN_HOME/servers/WLS_REPORTS/logs/WLS_REPORTS.log
The servlet logs initially showed:
REP-65003: NetworkUtility:getIOR The request timed out
This suggested that the servlet could not obtain a valid Reports Server IOR, and at first glance it looked like a multicast or networking issue.
One advantage we had during the investigation was the availability of a working Oracle Forms and Reports environment where Reports Server was starting successfully. This allowed us to compare the healthy and failing environments before moving deeper into diagnostics.
To validate the networking theory, we compared the following operating system level settings on both systems:
ip route ip link ip maddr show sysctl -a | grep rp_filter sysctl -a | grep multicast
We also compared the Reports networking configuration between the working and failing environments:
$DOMAIN_HOME/config/fmwconfig/components/ReportsServerComponent/rep_server1/rwnetwork.conf
To validate actual multicast traffic, packet captures and network traces were collected from both systems. The objective was to confirm:
- UDP port 14021 binding
- Multicast group registration
- Packet transmission
- Packet reception
The following commands were used:
Verify multicast group membership ip maddr show Capture Reports multicast traffic tcpdump -ni <interface> udp port 14021 -vvv Capture traffic to the configured multicast group tcpdump -ni <interface> host 228.5.6.7 -vvv
The traces confirmed that UDP port 14021 was successfully bound, the multicast group was successfully joined, and Reports discovery packets were being transmitted.
The comparison with the working environment did not reveal any meaningful networking difference capable of explaining the startup failure. In other words, the communication channel itself appeared to be operational.
Although the servlet continued to report:
REP-65003: NetworkUtility:getIOR The request timed out
The evidence suggested that this was likely a downstream symptom rather than the root cause.
At this point, the investigation shifted back to the Reports Server startup process itself.
The more useful information came from the Reports Server diagnostics. The server log showed that startup was progressing much further than expected:
[reports] [TRACE:32] [REP-56025] RWServer:startServer Reports Server is starting up. [reports] [NOTIFICATION:16] [REP-65000] Multicast:init Communication channel is initialized. [reports] [NOTIFICATION:16] [REP-56023] ServerImpl:ServerImpl Idle thread has been started. [reports] [NOTIFICATION:16] [REP-56020] EngineManager:spawnEngine Launching engine rwEng-0. [reports] [NOTIFICATION:16] [REP-56021] EngineManager:spawnEngine Engine rwEng-0 has been launched. [reports] [NOTIFICATION] [REP-56105] EngineManager:manage Engine rwEng-0 died with error.
This log changed the entire direction of the investigation.
Until this point, the focus had been on servlet communication and multicast discovery because of the repeated:
REP-65003: NetworkUtility:getIOR The request timed out
However, the Reports Server diagnostics clearly showed that:
- Reports Server startup was successful
- Multicast initialization was successful
- Engine creation was attempted successfully
- The failure occurred only after rwEng-0 was launched
In other words, the problem was no longer:
- Servlet
- Network
- Multicast
- Reports Discovery
The problem had now moved to:
Reports Engine (rwEng-0)
Investigating the Reports Engine Failure
At this point, the focus shifted entirely to the Reports Engine.The obvious question was:
Why is rwEng-0 dying immediately after startup?
Normally, if the engine fails because of a configuration issue, invalid parameter, or missing dependency, the error is usually captured in the Reports diagnostic logs.
However, in this case the engine was dying so quickly that no meaningful engine diagnostics were being generated.
This raised the possibility that the engine was not shutting down gracefully, but crashing.
To validate this theory, we searched for JVM crash artifacts:
find $DOMAIN_HOME -name “hs_err_pid*.log”
Multiple crash files were found:
$DOMAIN_HOME/hs_err_pid596994.log $DOMAIN_HOME/hs_err_pid596933.log $DOMAIN_HOME/hs_err_pid596913.log
The presence of these files immediately changed the nature of the investigation.
Unlike normal Reports errors, hs_err_pid files are generated only when the JVM encounters a fatal native crash.
Reviewing one of the files revealed:
# A fatal error has been detected by the Java Runtime Environment # SIGSEGV (0xb) # # Problematic frame: # C [libde.so.0+0x11a146] depcxfree+0x996 # # java_command: # oracle.reports.engine.RWEngine
This confirmed two important facts:
- The failing process was the Reports Engine (rwEng-0).
- The crash was occurring inside Oracle native code (libde.so.0).
The investigation moved away from Reports configuration to native JVM crash analysis.
Is libde.so.0 Really the Problem?
Every JVM crash file was pointing to the same native library and function:
libde.so.0 ==> depcxfree
The obvious assumption was that either the library itself was corrupted, incorrectly patched, or different from the working environment.
Since we already had a healthy Forms and Reports environment available for comparison, the first step was to verify whether both environments were running the same middleware patch level.
Patch inventories were compared using:
$ORACLE_HOME/OPatch/opatch lsinventory
During the comparison, we discovered that the working environment had WLS PSU applied, whereas the failing environment did not.
To eliminate that possibility, the same WebLogic PSU was applied to the failing environment.
After patching, Reports Server was tested again.
Unfortunately, the behaviour remained unchanged.
The engine continued to fail with the same startup sequence and the same JVM crash files continued to be generated:
Launching engine rwEng-0 Engine rwEng-0 has been launched Engine rwEng-0 died with error
With middleware patch levels now aligned, attention shifted back to the native library itself.
The following comparisons were performed between the working and failing environments:
find $ORACLE_HOME -name "libde.so*" sha256sum $ORACLE_HOME/lib/libde.so.0 ls -l $ORACLE_HOME/lib/libde.so.0 ldd $ORACLE_HOME/lib/libde.so.0
Since the crash was consistently occurring inside libde.so.0, we even copied the library directly from the working environment to the failing environment and restarted Reports Server.
Surprisingly, the result was exactly the same.
The Reports Engine continued to crash with:
SIGSEGV Problematic frame: C [libde.so.0] depcxfree
It became clear that although the crash manifested inside libde.so.0, the library was probably not the root cause. Rather, the failure was being triggered elsewhere during engine initialization.
The next logical question became:
What is calling libde.so.0 and why?
To answer that question, the investigation moved from JVM crash analysis into native process debugging using gdb.
Using GDB to Capture the Native Crash
Since the crash was now clearly happening inside the Reports Engine startup path, we needed a way to capture the native failure directly.
For that purpose, we temporarily modified $DOMAIN_HOME/reports/bin/rwengine.sh and launched the engine through gdb debugger.
The objective was to allow the JVM to complete its normal startup sequence, stop when the Reports native code path was reached, and capture the native stack at the moment of failure.
The following debugging block was added to rwengine.sh:
GDBCMDS=/tmp/rwengine_gdb_$$.cmd GDBLOG=/tmp/rwengine_gdb_$$.log cat > "$GDBCMDS" <<'GDBEOF' set pagination off set confirm off set breakpoint pending on set print thread-events off handle SIGSEGV nostop noprint pass handle SIGBUS nostop noprint pass handle SIGILL nostop noprint pass break depcxcreate commands silent printf "\n\n*** Hit depcxcreate in libde/libde.so.0. Now enabling stop on SIGSEGV. ***\n\n" handle SIGSEGV stop print nopass handle SIGBUS stop print nopass handle SIGILL stop print nopass continue end run printf "\n\n*** GDB stopped. Capturing diagnostic information. ***\n\n" info program bt bt full info sharedlibrary thread apply all bt GDBEOF exec gdb -q -x "$GDBCMDS" \ --args $ORACLE_HOME/oracle_common/jdk/jre/bin/java $SERVERMODE $SERVERINFO \ > "$GDBLOG" 2>&1
The resulting stack trace looked like this:
Java_oracle_reports_engine_EngineImpl_CInitEngine rxinit rxinita_reports ropint ropintcp depcccreate depcxcreate depcxfree SIGSEGV
The failure was occurring at CInitEngine before the Reports Engine could even complete initialization.
However, despite having the native stack trace, we still did not know why the crash was happening.
At this stage, we knew where the engine was crashing, but the actual trigger remained unknown.
The next objective was to understand what the engine was doing immediately before the crash occurred.
Using LD_DEBUG to Understand Engine Initialization
The GDB trace had already confirmed that the Reports Engine was crashing during native initialization inside CInitEngine. However, it still did not explain why the crash was happening.
Till now, we knew where the crash was occurring, but we did not know what was causing it.
The next step was to observe the Reports Engine startup process in greater detail. To do that, we enabled LD_DEBUG and traced native library activity during engine initialization. For that, we again temporarily modified:
$DOMAIN_HOME/reports/bin/rwengine.sh
The change was added immediately before the final Java execution line.
export LD_DEBUG=libs,files,versions export LD_DEBUG_OUTPUT=/tmp/rwengine_lddebug_$$ exec $ORACLE_HOME/oracle_common/jdk/jre/bin/java $SERVERMODE $SERVERINFO \ > /tmp/rwengine_stdout_$$.log 2>&1
This was done only for troubleshooting and not as a permanent change.
This step gave us the first meaningful Oracle-side clue. The debug output showed:
dbgc_init_all failed with ORA-48141
That was a major breakthrough because it shifted the focus away from the native crash symptom and toward Oracle diagnostic initialization.
We now knew that the Reports Engine was failing during the initialization of its diagnostic repository, but the exact OS-level operation causing the failure was still unclear. To pinpoint the issue, we turned to strace, which allowed us to identify the specific directory creation operation that was failing.
The Breakthrough – strace Finally Revealed the Real Failure
The LD_DEBUG output gave us the first Oracle-level clue:
dbgc_init_all failed with ORA-48141
We still needed to identify the exact operating system call that was failing. To do that, we modified $DOMAIN_HOME/reports/bin/rwengine.sh and traced the engine startup using strace.
To capture the failing operation, we added the trace at the final execution step, just before the Java process was launched:
TRACE_BASE=/tmp/rwengine_ora48141_trace_$$ OUT_LOG=/tmp/rwengine_ora48141_stdout_$$.log echo "===== rwengine ORA-48141 trace started at $(date) =====" > "$OUT_LOG" echo "PID=$$" >> "$OUT_LOG" echo "USER=$(id)" >> "$OUT_LOG" echo "PWD=$(pwd)" >> "$OUT_LOG" echo "ORACLE_HOME=$ORACLE_HOME" >> "$OUT_LOG" echo "DOMAIN_HOME=$DOMAIN_HOME" >> "$OUT_LOG" echo "ORACLE_BASE=$ORACLE_BASE" >> "$OUT_LOG" echo "TMP=$TMP TEMP=$TEMP TMPDIR=$TMPDIR" >> "$OUT_LOG" echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >> "$OUT_LOG" echo "SERVERINFO=$SERVERINFO" >> "$OUT_LOG" exec strace -ff -tt -T -s 2000 \ -e trace=file,process,signal,write \ -o "$TRACE_BASE" \ $ORACLE_HOME/oracle_common/jdk/jre/bin/java $SERVERMODE $SERVERINFO \ >> "$OUT_LOG" 2>&1
This time, strace clearly showed the failing location.
The key output was:
stat("/application/oradiag_op_1426/diag/plsql/user_op_1426/host_2005339290_82") = -1 ENOENT (No such file or directory)
mkdir("/application/oradiag_op_1426", 0775) = -1 EACCES (Permission denied)
write(1, "dbgc_init_all failed with ORA-48141\n", 36) = 36
This confirmed that the engine was failing while trying to create its diagnostic directory structure.
Until this point, the crash had only appeared as a JVM segmentation fault inside libde.so.0. strace gave us the exact OS-level failure behind it.
The Reports Engine was trying to create:
/application/oradiag_op_1426
but the operating system denied the request with:
EACCES (Permission denied)
That became the real root cause behind the engine startup failure.
Why Was Oracle Using /application?
The strace output finally revealed the failing directory:
/application/oradiag_op_1426
The next obvious question was:
Why is Oracle trying to write under /application?
After reviewing the engine environment and startup configuration, we found that ORACLE_BASE was not explicitly defined for the Reports Engine runtime.
When Oracle diagnostic initialization could not determine a valid Oracle base location, it fell back to the application user’s HOME directory:
HOME=/application
Unfortunately, /application was an application user home directory but user did not have permission to create directories underneath it.
As a result, Oracle attempted mkdir(“/application/oradiag_op_1426”, 0775) which failed with EACCES (Permission denied)
This failure propagated back into Oracle diagnostic initialization and resulted in dbgc_init_all failed with ORA-48141 which ultimately caused the Reports Engine JVM to crash during native initialization.
Solution provided
The fix turned out to be surprisingly simple.
We explicitly defined a valid writable Oracle base location:
export ORACLE_BASE=/weblogic/orabase
After updating the environment and restarting Reports Server:
- ORA-48141 disappeared
- No new hs_err_pid files were generated
- rwEng-0 remained running
- showjobs worked normally
The issue was fully resolved.
Final Thoughts
What started as:
REP-52266: The In-Process Reports Server Failed to Start
Initially looked like a Reports discovery problem.
The investigation then moved through:
REP-65003
↓
Multicast validation
↓
Reports Engine failure
↓
JVM crash analysis
↓
Native library investigation
↓
GDB debugging
↓
LD_DEBUG tracing
↓
strace analysis
↓
ORA-48141
↓
ORACLE_BASE
The most important lesson from this incident is that the first visible error is not always the real problem.
The servlet was reporting communication failures, the JVM was crashing inside a native library, and the stack traces pointed toward Oracle internals. Yet the actual root cause was a missing environment variable that caused Oracle ADR initialization to fall back to an unwritable location.