Slow Aux Copy since few weeks

Last post 01-13-2014, 11:04 AM by Mickael.LEDUC. 4 replies.
Sort Posts: Previous Next
  • Slow Aux Copy since few weeks
    Posted: 01-08-2014, 5:19 AM

    Hello And happy new user to all CommVault Forum User,

     

    I'm here, because, I have currently a strange problem with our Simpana infrastructure.

    Since few weeks, it seems that our Aux Copy run slowly.

    We are using Simpana 9 SP10, and Aux Copy run over a 10 Gb/s LAN connection

    I had this problem before, but I have used Disk optimized option instead of Netword Optimize in order to get better performance. And It was very impressive.

     

    I m also using the registry key Look Ahead. But I don't see a real improvement when I set this registry key !

     

    In order to identify the cause of my problem, I run a report on Aux copy.

    I get this into Excel, and draw some graphic.

    The first one show the throughput of the auxcopy. We can sse the improvement of Disk optimze parameters which was setting on 1st november.  Then on 1st December we can see that the throughput decrease.

     

    On the second graphic, we can the the application size, which is still the same. We can the Data on Network, which is still the same too, and we can see the Data Written, which change on 1st Decembre.

    Befor 1st December, the Data written is very low, and then after that, the data written follow exactly all the Data on Network.

     

    Is it normal ?

    I'm sorry to annoy you with that problem, but i need some help here.

    Thank you very much


  • Re: Slow Aux Copy since few weeks
    Posted: 01-09-2014, 5:54 PM

    Hello Mickael,

    Actually, looking at your graph, it seems like the slowdown started closer to Nov 25th, not Dec 1st. Not that it matters much but maybe you already asked yourself what you could have changed in the environment on Dec 1st when you should probably ask yourself the same question but for Nov 25th or so.

    Which brings me to the same question, have you any clue if something on the CV side changed around Nov 25th? End of month usually means new service pack. Any chance this started with a new service pack install?

    I find it very interesting that your data written becomes the same as data transferred over the network at that time. I just looked at my stats and my data transferred is always pretty much the same as my data written which makes sense since it should be doing source side dedup (DASH copy) and only send over the network what needs to be written, nothing more. It is actually very odd that you had better throughput when you were sending more data over the network than necessary. It also doesn't seem to make sense that you suddenly need to write to disk so much more data than before though it could explain the slowdown as if your target maglib isn't fast enough, your aux copy might now be waiting for the maglib to write data to disk while it didn't need to before.

    Also, how are you calculating your throughput? I'm assuming your data comes from the CommCellAuxCopyInfo view in the database using columns ElapsedTime, NWTransBytes, DataWritten and ApplicationSize? And then dividing your application size by elapsed time?

    Do you have this issue on all your storage policies (or global DDB) or only for a specific one?

    I know there isn't much answer for now but I'm just trying to understand what could be happening and also trying to figure out what possible option you could have used to send way more data over the network then necessary and yet get a better throughput that way.

    A sealed DDB using store priming would have made sense but then, your 2 graph lines (data trans/written) would have been inverted so it doesn't work in this case...

    I'll keep thinking, shoot any info that might help!

    Phil

  • Re: Slow Aux Copy since few weeks
    Posted: 01-10-2014, 3:49 AM

    Hello Philippe,

     

    Thanks a lot for your post. It is very helpful.

     

    So It is very interessting, that you have looked for your value about Data on network and Data Written.

    As you say, I think that it's normal that all data transmitted on the network were written, as the aux copy work with source dedup !

    The strange fact is the beginning of the charts !

    I  haven't find yet the change that I have made on the end of november !

    Currently, I have just made a report of all aux copie, redirected to csv output. Then I have created the chart with excel !

     

    It seems that the problem occur on all the DDB, and aux copy. It's more difficult to identify the problem on the other AuxCopy because there less data to be copied.

    Currently, we have a call with the CommVault Support.

    We have run IOMeter tests on our DDB Storage, and it seems that the storage is'nt suitable . We get 400 IOPS, and we need to get 800 IOPS belong the CommVault Documentation !

     

    So we need to improve our storage performance.

    I think that before the first backup, I have already run the same test, and I had no problem with IOPS !

    But the strange things are :

    - It seems that the perform decrease quickly, so I think it should be a wrong parameter !

     - The first test I have run with IO meter was better than now ....

     

    So I need to investigate about that.

    I'll take inform about news !

    Thanks again for your posts , it was very helpfull !

     

    Mickael

     

  • Re: Slow Aux Copy since few weeks
    Posted: 01-10-2014, 3:28 PM

    Well, even though the difference between data transferred and data written is quite strange, it might not have anything to do with your issue.

    400 IOPS for a DDB drive is definitely on the low side. What type of disk are you using? If you've seen a sudden drop, would it be possible then that the server team modified their antivirus setting and it is now scanning your DDB? That could explain a sudden drop in performance. Do you have any idea what the IOPS was like when you ran IOMeter the first time?

    Phil

  • Re: Slow Aux Copy since few weeks
    Posted: 01-13-2014, 11:04 AM

    Hello Philippe,

     

     

     

    You're right, 400 IOPS it's quite low !

    We are using SATA 2 TB 7200RPM for deduplication data, and SAS  GB 15000 RPM for DDB data.

     

    I ve check the antivirus real time scan parameters, It's good !

     

    I think that before the performance drop, we have something like 1200 IOPS on SAS Disk, and 2000 IOPS on SATA Disk. I think that we get better performance on SATA Disk, because we have more disk in each Raid Group.

     

    Thanks again Philppe for your Help.

     

    Mickael

     

The content of the forums, threads and posts reflects the thoughts and opinions of each author, and does not represent the thoughts, opinions, plans or strategies of Commvault Systems, Inc. ("Commvault") and Commvault undertakes no obligation to update, correct or modify any statements made in this forum. Any and all third party links, statements, comments, or feedback posted to, or otherwise provided by this forum, thread or post are not affiliated with, nor endorsed by, Commvault.
Commvault, Commvault and logo, the “CV” logo, Commvault Systems, Solving Forward, SIM, Singular Information Management, Simpana, Commvault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, SnapProtect, ROMS, and CommValue, are trademarks or registered trademarks of Commvault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.
Close
Copyright © 2019 Commvault | All Rights Reserved. | Legal | Privacy Policy