With the preparations all done it’s time to complete the live upgrade! This comes with a number of complications that the simulated upgrade couldn’t replicate; live production environments are rarely as clean cut as virtual environments because they’re being used for a range of purposes, often with hidden quirks that have come about from years of system tweaking! This means finding a time when the customer is happy for us to shut down production environments which inevitably leads to a couple of night shifts when the systems are quiet! With the times arranged, a couple of customer engineers on standby, and full system backups run and confirmed by the customer it was time to get the upgrade underway!
We knew from the simulation upgrade that the most complicated section would be ArcSight ESM, so we made a plan with the customer to complete the full ESM upgrade with hops from ESM 7.2 to 7.4 and finally to 7.5 on the first night and then upgrade ArcMC and SmartConnectors on the second night. Both upgrades also included RHEL OS updates which added another layer to the plan. This was all translated to the customer engineers while the process was ongoing to make sure they were always in the loop and could request changes to the plan at any point.
As with the preparation upgrade, the start of the upgrade meant backing up the opt/arcsight folder! We went through the same process again but this ended up being too slow in backing up the data. After a consultation with the customer engineers a new disk was created in Azure to hold the backup and after this brief road bump away we went! These upgrade hops also involved an OS update, meaning we almost ran out of time, finishing at almost exactly 12am after sending an update to the customer engineers so they could see exactly what had been done.
Night two went a bit more smoothly with the upgrade of ArcMC and the SmartConnectors. Due to the nature of the ArcSight portfolio there was fewer steps for these products, even with a couple of additional hops during the upgrade. Again, we had to run an OS upgrade which ended up causing more issues than previously experienced! As is the way with live environments, there was a quirk with the customers’ system OS which meant no matter what we tried it would not be a quick process to run the update. After some consultation with the customer engineers it was agreed that we would leave the OS where it was and they would run the OS updates internally, with our support if required, once it had been established what was causing the errors to occur. With this decision made we ran the upgrade to ArcMC and established that everything had updated correctly before running the SmartConnector upgrades through the console. Again, we sent evidence of changes to the customer engineers and cleaned up the upgrade files on the customer system.
All in all these upgrades ran as smoothly as could be hoped, with a couple of bumps on the way but no major issues! With this being my first ArcSight upgrade experience it was definitely a steep learning curve but definitely something all ArcSight customers should be running to make sure they have the best protection for their company as possible! We have another upgrade coming on a different customers’ ArcSight products soon so we will be taking the lessons learned this time around to the next round!