D365 Finance & Operations and Dynamics AX Forum

Expand all | Collapse all

AOS crashing

  • 1.  AOS crashing

    Posted Aug 14, 2019 09:56 PM
    On Monday our AOS started to crash several times per day. This seems to happen primarily while running sales reports such as quote and sales  confirmation journals We engaged our partner and Microsoft support team. MS support has been quite slow in responding. When they finally did they suggested full compile, full CIL,etc. We complete that and then started testing reports with just 1-2 users in the system. We ran many reports for 20 minutes and then the AOS crashed again.

    I have this message in the error log.

    Object Server 01:
    [Microsoft][SQL Server Native Client 11.0][SQL Server]Could not find stored procedure 'MTI_DYNAX12R3_PROD_01_model.dbo.XU_IsModelIdInstalled'.
    { CALL [MTI_DYNAX12R3_PROD_01_model].[dbo].[XU_IsModelIdInstalled](?, ?) }
    session 1 (-AOS-)

    Any thoughts or suggestions? We submitted dump file and error logs to MS and are now playing the waiting game.

    We are on 2012 R3 which had been stable for a long time until Monday. Single AOS environment.

    ------------------------------
    Scott Yergensen
    MTI, Inc.
    Salt Lake City UT
    ------------------------------
    Conference-AXUG_200x200


  • 2.  RE: AOS crashing

    TOP CONTRIBUTOR
    Posted Aug 15, 2019 08:35 AM
    This seems obvious,  but what changed on Monday?   That error looks either network, or SQL permissions related.   The AOS is trying to access a stored procedue on the database and cannot.
    1. Does that stored procedue exist in your AX Model Database?
    2. What do the permissions for that SP look like?  The AOS service account needs access.

    The fact the AOS runs the report a number of times then fails,  makes me think the error is not strictly related to that report.  If you run no reports for 20 minutes,  then run one does it crash?

    ------------------------------
    Corey Vantilborg
    ERP Analyst
    Tigercat International Inc.
    Brantford ON
    ------------------------------

    Conference-AXUG_200x200


  • 3.  RE: AOS crashing

    Posted Aug 15, 2019 09:48 AM
    Edited by Jeff Austin Aug 15, 2019 09:49 AM
    Be sure to check the timing of that message closely.  I see that one when the AOS starts and verified with a restart in one of our non-prod environments.


    Do you see anything a little earlier in the log?  AOS crashes are pretty distinct in the event viewer.

    Jeff

    ------------------------------
    Jeff Austin
    Application Specialist 5
    Bush Brothers & Company
    Knoxville TN
    ------------------------------

    Conference-AXUG_200x200


  • 4.  RE: AOS crashing

    Posted Aug 15, 2019 10:45 AM
    Our AOS has been up for about 14 hours now. We are only about an hour into our business day. I expect it to crash still but am grateful for a bit of stability.

    Corey, Early on Monday our partner imported a "simple" customization that would check for duplicate customer PO requisition number and return an infolog telling user if it was the same. We tested in TEST environment without issue prior to deploying. The problem started two hours later. We instructed them to remove the code by noon on Monday but the problem persists.

    Jeff, I went back and checked the error log against my notes when we were testing. This is the message at the exact time when the AOS crashed.

    The Microsoft Dynamics AX Object Server 6.3$01-MTI_DYNAX12R3_PROD_01 service terminated unexpectedly. It has done this 4 time(s).

    System
    - Provider
    [ Name] Service Control Manager
    [ Guid] {555908d1-a6d7-4695-8e1e-26931d2012f4}
    [ EventSourceName] Service Control Manager
    - EventID 7034
    [ Qualifiers] 49152
    Version 0
    Level 2
    Task 0
    Opcode 0
    Keywords 0x8080000000000000
    - TimeCreated
    [ SystemTime] 2019-08-15T00:58:35.815233700Z
    EventRecordID 188053
    Correlation
    - Execution
    [ ProcessID] 748
    [ ThreadID] 6604
    Channel System
    Computer
    Security
    - EventData
    param1 Microsoft Dynamics AX Object Server 6.3$01-MTI_DYNAX12R3_PROD_01
    param2 4
    41004F005300360030002400300031000000


    ------------------------------
    Scott Yergensen
    MTI, Inc.
    Salt Lake City UT
    ------------------------------

    Conference-AXUG_200x200


  • 5.  RE: AOS crashing

    TOP CONTRIBUTOR
    Posted Aug 15, 2019 11:01 AM
    This is going to sound like I'm preaching a bit, and I apologize for that.

    A couple of things concern me your reply.
    1. It sounds like your partner used an XPO to import code into your production system.  If this is the case it is unsupported and unsafe.  The only safe way to get new code into AX2012 is via a ModelStore import and deploy process.  If a full modelstore import was done,  do you know the process?  Was a DB Sync completed before any users logged in?
    2. Don't trust your partner on this,  it is your environment.  
    3. I don't believe in coincidences, that imported code is probably the source of your issue. 

    Do you have modelstore back-ups? Or backups of the Model database?   I would be restoring the model database to a known good state.

    Finally,  when the crash happens, it is certain users only? If that is the case clearing Usage Data and their .AUC cache files may resolve it.  You could also clear the AOS caches,  but the server restart should have done that already.


    ------------------------------
    Corey Vantilborg
    ERP Analyst
    Tigercat International Inc.
    Brantford ON
    ------------------------------

    Conference-AXUG_200x200


  • 6.  RE: AOS crashing

    Posted Aug 15, 2019 11:45 AM
    Corey, Thank you for the response. Even if you are preaching it is 100% justified in this case. I am quite disappointed in the partner and myself for allowing this to happen. AX was already deployed when I started here and customizations already in place. We have not deployed many new customizations in my time here. It is clear to me that better procedures must be in place prior to any future customizations.

    1. I am checking with the partner to determine the method used. I will post back here shortly.
    2. Hard life lesson learned. Too much trust was placed in the partner. This will not happen again.
    3. No way in the world this was a coincidence.

    Microsoft engineer just asked for a meeting in 20 minutes so hopefully we will get somewhere on this shortly.

    ------------------------------
    Scott Yergensen
    MTI, Inc.
    Salt Lake City UT
    ------------------------------

    Conference-AXUG_200x200


  • 7.  RE: AOS crashing

    Posted Aug 15, 2019 11:29 AM
    If you have Premier/Unified support I'd start escalating with your TAM.  If you don't have support I'd start with your Account Exec.  We've had mixed results with support but I've found if you can get the issue raised within Microsoft they are pretty good to respond.

    I think a crash dump analysis could be the place to start.  I've seen some results come back pretty quickly once you can get someone on it.

    Get with your partner and determine the exact contents of their payload and verify that they in fact removed it in it's entirety (nothing sneaky hanging out in the USR layer, etc), performed a complete CIL compile (I always clear the xppil directories on each AOS) and DB synch, and restarted every AOS.

    I fully agree with Corey - we will only XPO into a non-prod environment - and - there are no such thing as coincidences.  Any update to PROD requires a downtime where we can perform a modelstore import and completely reboot the system.  A modelstore database restoration may well be in your future.

    ------------------------------
    Jeff Austin
    Application Specialist 5
    Bush Brothers & Company
    Knoxville TN
    ------------------------------

    Conference-AXUG_200x200


  • 8.  RE: AOS crashing

    TOP CONTRIBUTOR
    Posted Aug 15, 2019 11:53 AM
    Hi -

    Fargo picked up the support case.  You'll hear from one of the best crash experts soon - Larry.

    Thanks....Dave

    ------------------------------
    Dave Phillips
    Sr Support Escalation Engineer
    Microsoft
    Fargo ND
    ------------------------------

    Conference-AXUG_200x200


  • 9.  RE: AOS crashing

    TOP CONTRIBUTOR
    Posted Aug 15, 2019 11:58 AM
    Having been helped by Dave and Larry a number of times in the past,  you are in great hands.

    ------------------------------
    Corey Vantilborg
    ERP Analyst
    Tigercat International Inc.
    Brantford ON
    ------------------------------

    Conference-AXUG_200x200


  • 10.  RE: AOS crashing

    Posted Aug 15, 2019 12:32 PM
    Thank you. Just spoke to Larry. We are following his recommendations. I will update this as we learn more.

    ------------------------------
    Scott Yergensen
    MTI, Inc.
    Salt Lake City UT
    ------------------------------

    Conference-AXUG_200x200


  • 11.  RE: AOS crashing

    Posted Aug 15, 2019 06:39 PM
    Larry reviewed the dump files. He believes we had an issue with corrupted syslastvalue records. We reset usage data for all users per his suggestions during a break period for production this afternoon. So far we are stable. I will stress test tonight. I am grateful for the responses here and for getting assigned to Larry. It is refreshing when you land on someone who really knows their product. If I meet him at Summit I'm buying him a beer or coffee. Same with you Corey, Jeff and Dave.  Thanks again.

    ------------------------------
    Scott Yergensen
    MTI, Inc.
    Salt Lake City UT
    ------------------------------

    Conference-AXUG_200x200


If you've found this thread useful, dive deeper into User Group community content by role