WebSphere Commerce 7 | Solr | Preprocess failed to run | SolrCore 'collection1' is not available | Could not load config file "search/solr/home/collection1/solrconfig.xml"

After migrating custom Solr files and if you try to run Solr Preprocess, sometimes you might face the following exceptions in running di-preprocess command,

INFO: Solr server core MC_10001_CatalogEntry_Unstructured_en_US status is initialized.
Aug 5, 2016 1:51:45 PM com.ibm.commerce.foundation.dataimport.process.DataImportProcessorMain executeDIH
INFO: Solr server core MC_10001_CatalogEntry_Unstructured_es_ES status is initialized.
Aug 5, 2016 1:51:45 PM null null
INFO: CWPKI0027I: Disabling default hostname verification for HTTPS URL connections.
Aug 5, 2016 1:51:46 PM com.ibm.commerce.foundation.dataimport.util.DataImportHelper printException
INFO: Data import process was unsuccessful. An unrecoverable error has occurred.
Throwable occurred: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. Error 500: {msg=SolrCore 'collection1' is not available due to init failure: Could not load config file /opt/IBM/WebSphere/CommerceServer70/instances/live/search/solr/home/collection1/solrconfig.xml,trace=org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Could not load config file /opt/IBM/WebSphere/CommerceServer70/instances/live/search/solr/home/collection1/solrconfig.xml
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:827)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:317)
        at com.ibm.commerce.foundation.internal.server.services.search.servlet.solr.SolrSearchDispatchFilter.doFilter(SolrSearchDispatchFilter.java:100)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)
        ...........
        ...........
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:278)
        at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
        at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
        at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
        at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
        at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)
Caused by: org.apache.solr.common.SolrException: Could not load config file /opt/IBM/WebSphere/CommerceServer70/instances/live/search/solr/home/collection1/solrconfig.xml
        at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:530)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:597)

Ideally, there won't be any "collections1" folder in the WebSphere Commerce Solr files. This might have happened because couple of Solr configuration files are missed in Search folder of the WebSphere Commerce installation directory.

Check whether all the Seach configurations files are in place and it should resolve your issue.

WebSphere Commerce 7 | Storefront | Product Display Page | "Add to Cart" button failed to load | CMN0420E | CWXFR9010I

After publishing new store in the Extended Sites, all the prices are coming up fine. But, in the Product Display Page, "Add to Cart" button is not getting displayed and it is throwing the following exception,

[8/11/16 5:42:31:712 GMT] 00000069 CommerceSrvr  E com.ibm.commerce.command.ECCommandTarget executeCommand CMN0420E: The following command exception has occurred during processing: "java.lang.NullPointerException". java.lang.NullPointerException
        at com.ibm.commerce.price.commands.PriceRuleGetContractUnitPriceCmdImpl.getRuleExecutionCommand(PriceRuleGetContractUnitPriceCmdImpl.java:512)
        at com.ibm.commerce.price.commands.PriceRuleGetContractUnitPriceCmdImpl.executeRule(PriceRuleGetContractUnitPriceCmdImpl.java:419)
        at com.ibm.commerce.price.commands.PriceRuleGetContractUnitPriceCmdImpl.calculatePrices(PriceRuleGetContractUnitPriceCmdImpl.java:198)
        at com.ibm.commerce.price.commands.PriceRuleGetContractUnitPriceCmdImpl.performExecute(PriceRuleGetContractUnitPriceCmdImpl.java:123)
        ...........
        ...........
        at com.ibm.commerce.command.AbstractECTargetableCommand.execute(AbstractECTargetableCommand.java:199)
        at com.ibm.commerce.price.commands.AbstractGetPriceCmdImpl.execute(AbstractGetPriceCmdImpl.java:1306)
        at com.ibm.commerce.price.commands.AbstractGetPriceCmdImpl.executeNextInChain(AbstractGetPriceCmdImpl.java:1224)
        at com.ibm.commerce.price.commands.EnsureOffersExistGetContractUnitPriceCmdImpl.performExecute(EnsureOffersExistGetContractUnitPriceCmdImpl.java:220)
        ...........
        ...........
        at org.apache.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:263)
        at org.apache.jasper.el.JspValueExpression.getValue(JspValueExpression.java:100)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspx_meth_c_set_274(_CachedProductOnlyDisplay.java:16947)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspx_meth_c_when_32(_CachedProductOnlyDisplay.java:17015)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspx_meth_c_choose_20(_CachedProductOnlyDisplay.java:17304)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspx_meth_c_if_64(_CachedProductOnlyDisplay.java:17346)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspService(_CachedProductOnlyDisplay.java:3099)
        at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:99)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:307)
        at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:576)
        ..............
        ..............
[8/11/16 5:42:31:713 GMT] 00000069 ExtendedInfo  I   CWXFR9010I: Extended information : [URL=http://localhost/webapp/wcs/stores/servlet/ProductDisplay] [parameters=storeId[0]=10651 storeId[1]=10651 parent_category_rn=40722 productId[0]=5750001 productId[1]=5750001 urlRequestType=Base categoryId=40792 top_category=40718 catalogId[0]=12051 catalogId[1]=12051 langId[0]=-24 langId[1]=-24 ] [userId=-1002]
[8/11/16 5:42:31:715 GMT] 00000069 CommerceSrvr  E com.ibm.commerce.command.ECCommandTarget executeCommand CMN0420E: The following command exception has occurred during processing: "java.lang.NullPointerException". java.lang.NullPointerException
        at com.ibm.commerce.price.commands.AbstractGetPriceCmdImpl.createFromOutput(AbstractGetPriceCmdImpl.java:578)
        at com.ibm.commerce.price.commands.EnsureOffersExistGetContractUnitPriceCmdImpl.performExecute(EnsureOffersExistGetContractUnitPriceCmdImpl.java:222)
        at com.ibm.commerce.command.ECCommandTarget.executeCommand(ECCommandTarget.java:157)
        at com.ibm.ws.cache.command.CommandCache.executeCommand(CommandCache.java:332)
        at com.ibm.websphere.command.CacheableCommandImpl.execute(CacheableCommandImpl.java:166)
        at com.ibm.commerce.command.MeasuredCacheableCommandImpl.execute(MeasuredCacheableCommandImpl.java:63)
        at com.ibm.commerce.command.AbstractECTargetableCommand.execute(AbstractECTargetableCommand.java:199)
        at com.ibm.commerce.price.commands.AbstractGetPriceCmdImpl.execute(AbstractGetPriceCmdImpl.java:1306)
        at com.ibm.commerce.price.commands.AbstractGetPriceCmdImpl.executeNextInChain(AbstractGetPriceCmdImpl.java:1224)
        at com.ibm.commerce.price.commands.CompositeGetContractUnitPriceCmdImpl.executeCommand(CompositeGetContractUnitPriceCmdImpl.java:339)
        at com.ibm.commerce.price.commands.CompositeGetContractUnitPriceCmdImpl.performExecute(CompositeGetContractUnitPriceCmdImpl.java:230)
        .............
        .............
        at javax.el.BeanELResolver.getValue(BeanELResolver.java:65)
        at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:143)
        at org.apache.el.parser.AstValue.getValue(AstValue.java:102)
        at org.apache.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:263)
        at org.apache.jasper.el.JspValueExpression.getValue(JspValueExpression.java:100)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspx_meth_c_set_274(_CachedProductOnlyDisplay.java:16947)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspx_meth_c_when_32(_CachedProductOnlyDisplay.java:17015)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspx_meth_c_choose_20(_CachedProductOnlyDisplay.java:17304)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspx_meth_c_if_64(_CachedProductOnlyDisplay.java:17346)
        at com.ibm._jsp._CachedProductOnlyDisplay._jspService(_CachedProductOnlyDisplay.java:3099)
        at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:99)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
        ............
        ............

When we tried to validate Price Lists & Price Rules for the store, it looks fine. But, there were no default Customer Segment and Price Rules setup in Commerce Accelerator and after setting up them from Commerce Accelerator, it started coming fine.

Here are the steps to setup default Customer Segments and Price rules,
1. Login to Commerce Accelerator
2. Select "Merchandise" --> "Catalog Filter and Price Rule"
3. In the "Catalog Filter" tab, set property value for the Catalog filter number

4. In the "Price Rule" tab, set property value for the Price Rule

WebSphere Commerce 7 | Solr | Preprocess Command failed to run | ERRORCODE=-4229, SQLSTATE=null | SQLCODE=-803, SQLSTATE=23505 | TI_APGROUP_0

Sometimes, we might face issue of Solr Preprocess failed with the following exceptions and it is because of Data issue in Categories and Catalog entries,
com.ibm.commerce.foundation.dataimport.preprocess.CatalogHierarchyDataPreProcessor determineCatalogEntriesForCatalog(Connection connection, String catalogID)
INFO: The catalog with ID: 10001 contains 18769 catalog entries.
com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][4.19.49] Batch failure.  The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null
        at com.ibm.db2.jcc.am.kd.a(Unknown Source)
        at com.ibm.db2.jcc.am.Agent.endBatchedReadChain(Unknown Source)
        at com.ibm.db2.jcc.am.gp.a(Unknown Source)
        at com.ibm.db2.jcc.am.gp.c(Unknown Source)
        at com.ibm.db2.jcc.am.gp.executeBatch(Unknown Source)
        at com.ibm.commerce.foundation.dataimport.preprocess.CatalogHierarchyDataPreProcessor.populateTable(CatalogHierarchyDataPreProcessor.java:1271)
        at com.ibm.commerce.foundation.dataimport.preprocess.CatalogHierarchyDataPreProcessor.process(CatalogHierarchyDataPreProcessor.java:676)
        at com.ibm.commerce.foundation.dataimport.preprocess.CatalogHierarchyDataPreProcessor.process(CatalogHierarchyDataPreProcessor.java:448)
        at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.processDataConfig(DataImportPreProcessorMain.java:1072)
        at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.execute(DataImportPreProcessorMain.java:942)
        at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.main(DataImportPreProcessorMain.java:385)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:267)
Aug 9, 2016 2:21:04 PM com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain processDataConfig(DataProcessingConfig, String)
INFO: Error for batch element #172: DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, SQLERRMC=1;WCSDB.TI_APGROUP_0, DRIVER=4.19.49
Aug 9, 2016 2:21:04 PM com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain processDataConfig(DataProcessingConfig, String)
INFO: SQL:
Aug 9, 2016 2:21:04 PM com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain logExitCode
INFO:
The program exiting with exit code: 1.
Data import pre-processing was unsuccessful. An unrecoverable error has occurred.

This issue might have happend because one Category is mapped with multiple Categories or One Product might be mapped with multiple Categories. Here are the queries to find these entries,

select catentry_id, catgroup_id from catgpenrel where catentry_id in (select catentry_id from catgpenrel where catalog_id = 10001 group by catentry_id having count(catentry_id) > 1)

select catgroup_id_parent, catgroup_id_child from catgrprel where catalog_id = 10001 and catgroup_id_child in (select catgroup_id_child from catgrprel where catalog_id = <catalogId> group by catgroup_id_child having count(catgroup_id_child) > 1)

Once these Catalog entries and groups issue has been fixed, re run Preprocesss command and it should work fine.

WebSphere Commerce 7 | Local setup of WCS & IBM HTTP Server | Not able to hit local server | Request is not reaching to WAS

I got some weird issue in my local environment where WebSphere Commerce Server was able to start properly. But, I am not able open Storefront pages as the request is not reaching the Application Server.

Later, I have realized it was happened because IBM HTTP Server was not started.


After, WebSphere Commerce installation in the local environment and on top of it, IBM HTTP Server has been installed so that Static content will be served from Web Server instead of WCS Application Server as mentioned in the following document,
http://www.ibm.com/developerworks/library/co-config-httpserver-was/

As part of this IBM HTTP Server installation, you might have run Jython Script so that, Ports  80, 443 will be used by HTTP Server. Later, you will be able to hit only local WebSphere Commerce Server through IBM HTTP Server itself.

If we stop HTTP Server and try to hit Storefront, it will not reach the WebSphere Commerce Server. In the future, if you don't want to use IBM HTTP Server and directly, hit WCS Server itself, then update Jython Script "configureWCPorts.py".

AdminTask.modifyServerPort ('server1', '[-nodeName localhost -endPointName WC_defaulthost -host localhost -port 80 -modifyShared true]')

AdminTask.modifyServerPort ('server1', '[-nodeName localhost -endPointName WC_defaulthost_secure -host localhost -port 443 -modifyShared true]')

AdminTask.modifyServerPort ('server1', '[-nodeName localhost -endPointName WC_PORT_4 -host localhost -port 18006 -modifyShared true]')AdminConfig.save()

And run the following Jython Script as mentioned in the document,
wsadmin -lang jython -f configureWCPorts.py -conntype NONE

Now, you should be able to hit WebSphere Application Server directly with out IBM HTTP Server.

WebSphere Commerce 7 | Solr Search | Running Preprocess is failing | CWFDIH0001 | CWFDIH0002 | ERRORCODE=-4461, SQLSTATE=42815

After Solr setup, when we tried to run di-preprocess command, it failing with the following exception,
       
Table name:     TI_GROUPING_0
Fetch size:     500
Batch size:     500
Aug 3, 2016 9:23:59 AM com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain logExitCode
INFO:
The program exiting with exit code: 1.
Data import pre-processing was unsuccessful. An unrecoverable error has occurred.

Aug 3, 2016 9:23:59 AM com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain logEndDateAndTime
INFO: Data import pre-processing ended:Wed Aug 03 09:23:59 GMT 2016
Aug 3, 2016 9:23:59 AM com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain logEndDateAndTime
INFO: Data import pre-processing completed in 4.287 seconds.
Aug 3, 2016 9:23:59 AM com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain handleExecutionException
SEVERE: Exception message: CWFDIH0001: Data Import encountered a generic error. The error is CWFDIH0002: An SQL exception was caught. The following error occurred: [jcc][1091][10404][4.19.49] Invalid data conversion: Parameter instance 5711501;5711502 is invalid for the requested conversion. ERRORCODE=-4461, SQLSTATE=42815.. , stack trace: com.ibm.commerce.foundation.dataimport.exception.DataImportSystemException: CWFDIH0001: Data Import encountered a generic error. The error is CWFDIH0002: An SQL exception was caught. The following error occurred: [jcc][1091][10404][4.19.49] Invalid data conversion: Parameter instance 5711501;5711502 is invalid for the requested conversion. ERRORCODE=-4461, SQLSTATE=42815..
        at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.processDataConfig(DataImportPreProcessorMain.java:1112)
        at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.execute(DataImportPreProcessorMain.java:942)
        at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.main(DataImportPreProcessorMain.java:385)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:267)
Caused by: com.ibm.commerce.foundation.dataimport.exception.DataImportSystemException: CWFDIH0002: An SQL exception was caught. The following error occurred: [jcc][1091][10404][4.19.49] Invalid data conversion: Parameter instance 5711501;5711502 is invalid for the requested conversion. ERRORCODE=-4461, SQLSTATE=42815.
        at com.ibm.commerce.foundation.dataimport.preprocess.AbstractDataPreProcessor.insertColumns(AbstractDataPreProcessor.java:284)
        at com.ibm.commerce.foundation.dataimport.preprocess.StaticAttributeDataPreProcessor.process(StaticAttributeDataPreProcessor.java:365)
        at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.processDataConfig(DataImportPreProcessorMain.java:1072)
        ... 7 more
Caused by: com.ibm.db2.jcc.am.SqlSyntaxErrorException: [jcc][1091][10404][4.19.49] Invalid data conversion: Parameter instance 5711501;5711502 is invalid for the requested conversion. ERRORCODE=-4461, SQLSTATE=42815

From the log file, we have noticed two catentries - 5711501;5711502 and we tried validate from the database, one Catentry is parent to another catentry. But where as both catentries has CATENTTYPE_ID has 'ProductBean' itself.

We have corrected the Catentries and rerun the di-preprocess, it was successful

WebSphere Commerce 7 | Storefront | Failed to compile JSP | SRVE0293E | JSPG0049E | Exceeding the 65535 bytes limit

Once the new code has been deployed to WebSphere Commerce environment and hit the Category Page, it starting throwing the following Exception,

[8/4/16 7:29:13:700 GMT] 00000059 webapp        E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[/DefaultStorefrontAssetStore/Layouts/SearchBasedCategoryPage.jsp]: com.ibm.ws.jsp.JspCoreException: JSPG0049E: /DefaultStorefrontAssetStore/Layouts/SearchBasedCategoryPage.jsp failed to compile :

/opt/IBM/WebSphere/AppServer/profiles/live/temp/WC_myNode_node/server1/WC_live/Stores.war/DefaultStorefrontAssetStore/Layouts/_SearchBasedCategoryPage.java : 967 : The code of method _jspService(HttpServletRequest, HttpServletResponse) is exceeding the 65535 bytes limit

        at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.translateJsp(AbstractJSPExtensionServletWrapper.java:620)
        at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper._checkForTranslation(AbstractJSPExtensionServletWrapper.java:487)
        at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.checkForTranslation(AbstractJSPExtensionServletWrapper.java:340)
        at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:156)
        at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:374)
        at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
        at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:398)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:241)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
        at com.ibm.commerce.struts.ECActionServlet.processRequest(ECActionServlet.java:232)
        at com.ibm.commerce.struts.ECActionServlet.doGet(ECActionServlet.java:173)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
        at com.ibm.commerce.struts.ECActionServlet.service(ECActionServlet.java:743)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
       
To fix the issue, here are the steps,

• Login to WAS Administration Console
• Navigate to Servers --> Server Types --> WebSphere application servers
• Click on server1
• In Configuration Tab, Click on Container Settings --> Web Container Settings --> Web Container

  

• Click on Additional Properties --> Custom properties

 

•  Click on New and provide the custom property,

          Name : com.ibm.wsspi.jsp.disableResourceInjection
          Value: true

•  Restart the server