Showing posts with label NODE MANAGER. Show all posts
Showing posts with label NODE MANAGER. Show all posts

Ignoring the trusted CA certificate warnings when connecting to Node Manager using WLST nmConnect()

Symptoms

When using Weblogic Scripting Tool (WLST) nmConnect() to connect to the node manager, notice warnings are seen for unsupported certificates (after running setWLSEnv.cmd or .sh)


Connecting to Node Manager ...
CA certificate "CN=T-TeleSec GlobalRoot Class 3,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object:
1.2.840.113549.1.1.11.>
<Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=T-TeleSec GlobalRoot Class 2,OU=T-Systems Trust Center,O=T-Systems Enterprise Services GmbH,C=DE". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=GlobalSign,O=GlobalSign,OU=GlobalSign Root CA - R3". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "OU=Security Communication RootCA2,O=SECOM Trust Systems CO.\,LTD.,C=JP". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
<Notice> <Security> <BEA-090898> <Ignoring the trusted CA certificate "CN=KEYNECTIS ROOT CA,OU=ROOT,O=KEYNECTIS,C=FR". The loading of the trusted certificate list raised a certificate parsing exception PKIX: Unsupported OID in the AlgorithmIdentifier object: 1.2.840.113549.1.1.11.>
Successfully Connected to Node Manager.

Steps

Oracle Weblogic Server: Key WLST Node Manager Commands

The Node Manager helps you remotely control WebLogic Server instances. WLST Node Manager commands help you access the Node Manager features. Following are examples that explain how to use the most important WLST Node Manager commands, from a day-to-day operational standpoint.

Connect to Node Manager

Assuming the Node Manager is already running (for example, started from the Windows service), you need to connect to the Node Manager using the nmConnect command before you run any of the Node Manager WLST commands. Note that you must specify a domain name (wl_server in this example) when you connect to the Node Manager.

cd C:\MyOra\Middleware\wlserver_10.3\common\bin
C:\MyOra\Middleware\wlserver_10.3\common\bin>wlst.cmd

wls:/offline> nmConnect('weblogic', 'welcome1', 'localhost', '5556', 'wl_server',
'C:\MyOra\Middleware\wlserver_10.3\samples\domains\wl_server','ssl')
Successfully Connected to Node Manager.

Note that in a production environment, you must first execute the nmEntroll command to enroll the machine on which the Node Manager is running before executing the nmConnect command to connect to the Node Manager. By executing the nmEnroll command, you ensure that the Node Manager credentials are available to the Managed Servers that the Node Manager manages. You run the nmEnroll command only once on each machine in a WebLogic domain.

Oracle Weblogic Server: Stopping the Node Manager

The simplest way to shut down the Node Manager is to just close the command shell in which it runs. You can also invoke the WLST stopNodeManager command in the online or offline mode. The command stops a running Node Manager process. This will not work with the scripted version of Node Manager, though.

cd C:\MyOra\Middleware\wlserver_10.3\common\bin
wlst.cmd
wls:/offline> nmConnect('weblogic', 'welcome1', 'localhost', '5556', 'wl_server',
'C:\MyOra\Middleware\wlserver_10.3\samples\domains\wl_server','ssl')
wls:/nm/wl_server> stopNodeManager()

If you try to shut down the Node Manager with the stopNodeManager command when you haven't started the Node Manager with the startNodeManager command, you'll get the following error:

wls:/nm/wl_server> stopNodeManager()
Traceback (innermost last):
...
weblogic.management.scripting.ScriptException: weblogic.management.scripting.ScriptException:
Error occured while performing startNodeManager : Problem stopping
the Node Manager. : Disabled command: QUIT
Use dumpStack() to view the full stacktrace

However, you can successfully stop the Node Manager process, even if you haven't started the Node Manager with the startNodeManager command, provided you've specified the property QuitEnabled=true when starting the Node Manager. You can specify the QuitEnabled property in the nodemanager.properties file. Once you do this, you can start the Node Manager as a Windows service and stop the service remotely via WLST.

How to Upgrade the JDK Used by Oracle WebLogic Server UNIX installations to a Different Version


There are two different approaches that can be used to achieve the required goal. Both will be described briefly:
A) Installing a new JDK home, and using symbolic link to "rename" the existing and new JDK directories.
Following this approach, the JDK location originally used during the installation will not change and hence it has the minimal drawback that it might keep a versioned name as for example "$INSTALL_HOME/jdk_".
In any of the cases, this approach is simpler and less error prone than Approach B)

B) Installing a new JDK home directory and replacing the JAVA_HOME environmental variable in *** all *** the scripts referring to it.

This is the preferred approach if you want to do a partial upgrade (affecting to specific domains only, or not affecting to nodemanager), but its difficulty is around on how to know which files do refer to a JDK location.

The initial list of files to change will be :
* $INSTALL_HOME/wlserver_10.3/common/bin/commEnv.sh
* $INSTALL_HOME/user_projects/domains/<mydomain>/bin/setDomainEnv.sh
* $INSTALL_HOME/wlserver_10.3/common/nodemanager/nodemanager.properties
* $INSTALL_HOME/utils/bsu/bsu.sh
* $INSTALL_HOME/utils/quickstart/quickstart.sh
* $INSTALL_HOME/utils/uninstall/uninstall.sh
even that a more exhaustive way to determine them, will be to run the following command:
find . -type f -name "*.sh" -exec grep -il JAVA_HOME {} \;

WebLogic Server: Setting Custom Memory Values (Heap Size)


Setting Custom Memory Settings for the Admin Server

Memory settings for the Admin Server can be modified by editing the $MW_HOME\user_projects\domains\\bin\startWebLogic startup script. Add the MEM_ARGS parameter to override the setDomainEnv USER_MEM_ARGS value here:
MEM_ARGS="-Xms<value>m -Xmx<value>m"
export MEM_ARGS

Setting Custom Memory Settings for the Managed Server

There are four options available to set memory arguments for Managed Server instance. To edit memory values, implement one of the following:
  1. When starting a managed server through the $MW_HOME\user_projects\domains\\bin\StartManagedWebLogic script, you can add the memory arguments desired.
    1. First set the domain environment by calling setDomainEnv.
    2. Call StartManagedWebLogic <ManagedServerName> <AdminURL> -Xms<value>m -Xmx<value>m
      where <ManagedServerName> is your managed server to be started.
      is the Adminstration Server address, e.g., http://localhost:7001
      <value> is the desired heap size ie: -Xms256m -Xmx256m to get a fixed heap size
    3. You still will see something like this in the output:
      JAVA Memory arguments: -Xms512m -Xmx512m
      but that is overridden by the given arguments.

Smart Update Fails to Apply Patch: java.io.IOException: Unable to backup file


When attempting to install a patch using Smart Update (BSU), the following error occurs:
java.io.IOException: Unable to backup file <path>/weblogic_patch.jar to <path>/weblogic_patch.jar.bak
Verbose logging gives some more details:
java.util.concurrent.ExecutionException: com;bea.plateng.patch.PatchInstallationException: java.io.IOException: Unable to backup file <path>/weblogic_patch.jar

Cause

This is a common problem customers experience when they try to apply patches while the WLS domain is running. Because the domain is running, the jar which Smart Update needs to write to is locked, and this error occurs.

Solution

Please stop the WLS domain, and make sure there are no processes or services running which would be holding a lock on that jar file. If Node Manager is running, you should also stop that.
Once all those processes and/or services are stopped, please run Smart Update and apply the patch. It should work.

How to Start Node Manager from WLST with JSSE Enabled


To enable JSSE for Node Manager when starting it through WLST requires -Dweblogic.security.SSL.enableJSSE=true to be passed as an argument. 
See http://docs.oracle.com/cd/E23943_01/web.1111/e13707/ssl.htm#BABIJEJD for more information about how to enable JSSE in different contexts.


To enable this option in WLS 12.1.1 and earlier, please apply the patch for unpublished defect 14174803. This patch is not required in WLS 12.1.2 and higher where this functionality is already included.

This patch adds a functionality to pass JVM arguments when starting Node Manager. After applying the patch you can execute the below command:

startNodeManager(NodeManagerHome="/home/oracle/keshav/wls12/wlserver_12.1/common/nodemanager",jvmArgs="-Dweblogic.security.SSL.enableJSSE=true")

Executing the above should enable JSSE on Node Manager when it is started through WLST.

Node Manager Fails with JSSE SSL Configured at the Admin Server


After enabling JSSE on Admin server, Node Manager was throwing the following error and unable to start managed servers using Node Manager.
  SSL header was received from peer aubdc00-ofm03s - aa.bb.cc.dd during SSL handshake.>
javax.net.ssl.SSLHandshakeException: [Security:090476]Invalid/unknown SSL header was received from peer aubdc00-ofm03s - aa.bb.cc.dd during SSL handshake.


       at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
       at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertSent(Unknown Source)
       at com.certicom.tls.record.ReadHandler.fireAlert(Unknown Source)
       at com.certicom.tls.record.ReadHandler.getProtocolVersion(Unknown Source)
       at com.certicom.tls.record.ReadHandler.checkVersion(Unknown Source)
       at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
       at com.certicom.tls.record.ReadHandler.readUntilHandshakeComplete(Unknown Source)
       at com.certicom.tls.interfaceimpl.TLSConnectionImpl.completeHandshake(Unknown Source)
       at com.certicom.tls.record.ReadHandler.read(Unknown Source)
       at com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
       at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:452)
       at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:494)
       at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:222)
       at java.io.InputStreamReader.read(InputStreamReader.java:177)
       at java.io.BufferedReader.fill(BufferedReader.java:136)
       at java.io.BufferedReader.readLine(BufferedReader.java:299)
       at java.io.BufferedReader.readLine(BufferedReader.java:362)
       at weblogic.nodemanager.server.Handler.run(Handler.java:71)
       at java.lang.Thread.run(Thread.java:736)

Node Manager: Common Problems and Resolutions

Following are some of the exceptions/errors and resolutions.

Host name verification (Node Manager log)

Following problem is due to Node manager Setup and seen in the node manager log:

<May 3, 2005 1:00:45 PM EDT> <Error> <NodeManager@xxxx11:5559> <NodeManager is not configured to receive commands from host : /10.62.3.215. Please update the trusted hosts file : /home/rbabu/nodemanager.hosts of the node manager by adding the hostname or ip address of /10.62.3.215>

Resolution: Add the host name or IP address to nodemanager.hosts and restart the node manager.
If, after adding the entry to the nodemanager.hostsfile you still see the error, add the following to the node manager start script and admin server.


Node Manager:

-Dweblogic.nodemanager.sslHostNameVerificationEnabled=false
Admin Server:

-Dweblogic.security.SSL.ignoreHostnameVerification=true
Or you can do the same using console as shown below:
Under Keystores & SSL tab, click on "Advanced Options." Change the Hostname verification to None.

Checklist for Troubleshooting Node Manager SSL Problems

  1. Check what certificates are being used. Demo, Commercial, self-signed?
  2. In the case of demo certificates make sure none of the settings are changed. You don't need any entries in the nodemanager.propertiesfile. Nor you do not need to make any changes to the settings in the admin or managed server.
  3. In the case of commercial certificates (Verisign, Thawte, Comodo, etc.) make sure that the certificate chain is complete and the root and intermediate certificates are configured properly.
  4. If the certificates are self-signed, make sure you have followed the sequence mentioned earlier in this document.
  5. Make sure that the validation dates are correct.
  6. Turn on the debug flags on both admin server and managed server to get all possible information.
  7. Node manager debug flag. This flag needs to be added to the start script
  8. (The log files are located under <NodeManagerHome>/nodemanager.log)

  • -Dssl.debug=true -Dweblogic.StdoutDebugEnabled=true
  • -Dweblogic.StdoutDebugEnabled=true -Dweblogic.nodemanager.debugEnabled=true -Dweblogic.nodemanager.debugLevel=90

WebLogic Admin Server Shows Stuck Threads Due To A Misconfiguration Of The Node Manager

The Admin Server goes to "Warning" state (stuck thread error) after a couple of hours, even though there are no activities in the deployed application Oracle Enterprise Repository (OER). When OER is shutdown, the Admin Server works fine. The Admin Server goes in warning state when the managed server is started and logs show the following error:


SERVER = AdminServer MESSAGE = [STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "610" seconds working on the request "weblogic.kernel.WorkManagerWrapper$1@1f9c1ae0", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-88 "[STUCK] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:149)
java.io.InputStream.read(InputStream.java:85)
com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
com.certicom.tls.record.ReadHandler.read(Unknown Source)
^-- Holding lock: com.certicom.tls.record.ReadHandler@1f9dbb78[thin lock]
com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:250)
sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:289)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:125)
^-- Holding lock: java.io.InputStreamReader@1f9c05d7[thin lock]
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:105)
java.io.BufferedReader.readLine(BufferedReader.java:288)
^-- Holding lock: java.io.InputStreamReader@1f9c05d7[thin lock]
java.io.BufferedReader.readLine(BufferedReader.java:362)
weblogic.nodemanager.client.NMServerClient.checkResponse(NMServerClient.java:287)
weblogic.nodemanager.client.NMServerClient.checkResponse(NMServerClient.java:312)
weblogic.nodemanager.client.NMServerClient.start(NMServerClient.java:93)
^-- Holding lock: weblogic.nodemanager.client.SSLClient@1f9c0e0d[thin lock]
weblogic.nodemanager.mbean.StartRequest.start(StartRequest.java:75)
weblogic.nodemanager.mbean.StartRequest.execute(StartRequest.java:45)
weblogic.kernel.WorkManagerWrapper$1.run(WorkManagerWrapper.java:63)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}

NodeManager not Reachable: java.io.IOException: Invalid State File Format

Weblogic managed Servers cannot be started via the WLS console as Node Manager is not reachable.
Also, the nodemanager.log file is showing the following messages:
domainName\servers\serveName\data\nodemanager\startup.properties">
FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName\servers\serveName\data\nodemanager\startup.properties">
<Feb 9, 2011 11:06:01 AM> <WARNING> <There was a problem initializing the domain 'steffworld' at '\FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName'. Please make sure that this domainName: 'domainName' is registered and is fully enrolled for this nodemanager at: '\FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName'.>
<Feb 9, 2011 11:06:01 AM> <WARNING> <I/O error while reading domain directory>
java.io.IOException: Invalid state file format. State file contents:
at weblogic.nodemanager.common.StateInfo.load(StateInfo.java:135)
at weblogic.nodemanager.server.ServerMonitor.loadStateInfo(ServerMonitor.java:475)
at weblogic.nodemanager.server.ServerMonitor.isCleanupAfterCrashNeeded(ServerMonitor.java:139)
at weblogic.nodemanager.server.ServerManager.recoverServer(ServerManager.java:255)
at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:103)
at weblogic.nodemanager.server.DomainManager.<init>(DomainManager.java:55)
at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:257)
at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:224)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:108)
at weblogic.nodemanager.server.Handler.run(Handler.java:70)
at java.lang.Thread.run(Thread.java:619)

Changes

After a power failure, the server machine (running as a Windows service) restarted automatically.

Cause

The state file of the managed server is in an invalid state: for example, it may be empty. Under each managed server directory, there is a NodeManager directory containing a state file <managed_server_name>.state. If this file is empty or corrupt, then the described errors occur.
For example, under \FMW\WLS1033\Oracle\Middleware\user_projects\domains\domainName\servers\<serverName>\data\nodemanager, the <serverName>.state file is empty.
The cause of the empty file could be one of the following:

WebLogic Server: Procedure for configuring Node Manager with SSL

Steps for configuring Node Manager with SSL for WebLogic Server.

First create custom certificates using the keytool command utility:

keytool -genkey -alias mykey -keyalg RSA -keysize 1024 -dname "CN=Tariq, OU=Customer Support, O=BEA Systems Inc, L=Denver, ST=Colorado, C=US" -keypass password -keystore identity.jks -storepass password
keytool -selfcert -v -alias mykey -keypass password -keystore identity.jks -storepass password -storetype jks
keytool -selfcert -v -alias mykey -keypass password -keystore identity.jks -storepass password -storetype jks
keytool -import -v -trustcacerts -alias mykey -file rootCA.der -keystore trust.jks -storepass password


Now configure "Custom Identity and Custom Trust" for admin and managed servers from console
Now enter the Key Alias and Private Key Passphrase under the SSL tab for both the servers from console