Vystar online banking debacle

[pierre]

Online banking has been down for a week (app and website) with no ETA for resolution.

The CEO was on vacation during the switch, and did not come back into town until this weekend.

Could this have been handled worse?

[BridgeTroll]

Add to that the vendor supplying the solution is a company partially owned by vystar...

https://www.cutimes.com/2021/04/27/vystar-credit-union-invests-20-million-in-nymbus-cuso/

[JPalmer]

A drive up ATM by my house has had a steady stream of cars this past week.  I guess you can check your balance and do a few other basic transactions.  I certainly check my balance on Mondays and would be annoyed.

[Steve]

I'd really love to know what happened here and I doubt we'll get that publicly. While not Financial systems, this sort of thing is what I deal with on a daily basis (system cutovers, integrations, performance, vendor management, etc.).

 - I know they delayed this multiple times, which gave me hope they were actually being thorough. Apparently not.
 - It looks like Brian Wolfburg already had vacation planned when they selected this date (as it was just announced publicly a few weeks back). This is tough - I'm not a CEO but I've been in this spot as the lead on major projects. On one hand the CEO isn't going to jump in a code the thing, but on the other hand sticking the poor Chief Customer Officer out there isn't ideal either (unless technology rolls up to him, which I doubt). It's like when there's a delayed flight the beating the gate agents get at the airport. The gate agent didn't cause this.
 - This is obviously a black eye for both VyStar and Nymbus....particularly Nymbus given unless this can't be fixed VyStar is going to keep members and use the system.....but if I'm a small-midsized bank looking at software this would have me concerned.

As an account holder, it's annoying but not catastrophic. Their phone system is working fine, as is all of their core systems it appears. All of my deposits that I expected last week hit exactly when I expected them to, so everything seems fine with flow, which does make some tech if I put my tech hat on. It's basically banking in 1999 lol.

Now, this does give me some concern about tech on a larger scale with VyStar. I have a checking/savings account with them (and a car loan), but no "serious" money (investments, IRA, etc.), Car Loan aside, all the money I have with them is NCUA insured and I personally have enough liquid in other places that if VyStar were to close up shop tomorrow and I needed cash while the NCUA figured out everything I'd be fine.

That said, I know not everyone at VyStar is in that position.

[WarDamJagFan]

My wife and I use Vystar for most of our daily transactions like paychecks and bills. Everything else finance related is outside of Vystar. I'll have to go by an ATM or Branch sometime this week as bills need to be paid and it's tough when you can't get a snapshot of one's account. But as someone who works directly with financial institutions and their mission critical software, this is an abysmal failure of planning and execution. Vystar's PR strategy the past week + is it's own epic fail.

Vystar made an interesting comment when interviewed that 'scalability' was one of the key drivers of the current outage. I'm no software engineer, but I know that beta testing a holistic change in one's platform is a monumental task that requires an extreme level of attention to detail. It's also odd that there was no way for Vystar to run their "old" platform in parallel with the new platform until all the kinks are worked out. Every client who's tried to do the cheap way out and cancel their contract with us so there's no overlap in service between old and new providers almost ALWAYS scrambles back asking for us to turn their service back on. Why? Because their workflows are slower than anticipated with the new software or it's simply not ready for full use thereby putting unnecessary risk on themselves and their underlying clients. Depending on the level of complexity a particular client has and if that client needed large amounts of customized engineering within the solutions they were subscribed to, it's no easy fix and can take a long time to bring them back online. So when Vystar says switching back to the old platform isn't something they can do overnight, that makes sense to me given the above scenario.

The current situation with Vystar, at least to me, screams that they were overconfident in Nymbus being fully operational by Day 1 thereby rendering any extension of their previous platform unnecessary. It also doesn't help when your CEO stays on Vacation - unavailable for public comment - while said business is in the middle of what is now the worst operational crisis in its entire existence.

[Steve]

^Not knowing how their system is architected I have no idea if parallel would be an option, but many sites HAVE done that.

To your point, they clearly haven't done any executive work with crisis management, because their public response is horrible. Sticking the poor Chief Customer Officer out there is terrible. His team is the one taking the brunt of this, and his team can't fix the core issue. I'll say my wife went by a branch Saturday and they apparently were extremely nice and accommodating, which was good.

It seems like they had no concept that after a day or so this WOULDN'T work.

[BridgeTroll]

Is it normal for an executive officer of the CU to also be on the board of directors of the primary vendor?

https://vystarcu.org/who-we-are/our-team-senior

https://www.nymbus.com/about/

[Lunican]

Interesting they can't roll this back. When I used to do this stuff we'd have a full backout plan that was practiced in a test environment dozens of times at a minimum. If at any point in the implementation things were not as expected we could execute the backout plan. It was not always easy technically or politically but a 7 day outage was not an option!

[jaxlongtimer]

I've been involved in a switchover.  I am not a techie but I have been told you typically can only maintain live data in one system or another, feasibly.  So, once you commit to the new system by going live, it's not like you can move all the updated data back to the old system.  Plus, there are integrations (i.e. connective links) with other systems that are not easy to move back and forth.  And, if you have a team familiar (having spent months or years on this project) with your situation that is best suited to fix things, they may only have the ability to focus on one system at a time.  I have read they moved to this system due to the old one aging out.  That may also make it hard to go back if support for the old system has faded away.
 
Essentially, once you commit to going live, it isn't easy turning back the clock, if you can do it all.  It may be better to put your limited resources into fixing the new system pronto.

As to the CEO being on the board of Nymbus, I don't think it is a conflict because I read Vystar actually owns a good sized chunk of Nymbus.  Thus, as a key investor, it would be appropriate for them to be represented on its board.  What is a double whammy is not only does Vystar take a PR or more hit for this issue, but it also may substantially devalue their investment in Nymbus, at least in the short run.

[Steve]

Is it normal for an executive officer of the CU to also be on the board of directors of the primary vendor?

https://vystarcu.org/who-we-are/our-team-senior

https://www.nymbus.com/about/

If they are a significant investor in that vendor, then it's not only appropriate but generally you wouldn't make the investment without the board seat. While I think the news should be covering this whole mess, I believe this angle is not where their energy should be.

[Steve]

Interesting they can't roll this back. When I used to do this stuff we'd have a full backout plan that was practiced in a test environment dozens of times at a minimum. If at any point in the implementation things were not as expected we could execute the backout plan. It was not always easy technically or politically but a 7 day outage was not an option!

This. I can't imagine back when I was at Fanatics if our ecommerce site was down for 7 days. We would probably have had an issue making payroll among other things.

[Steve]

I've been involved in a switchover.  I am not a techie but I have been told you typically can only maintain live data in one system or another, feasibly.  So, once you commit to the new system by going live, it's not like you can move all the updated data back to the old system.  Plus, there are integrations (i.e. connective links) with other systems that are not easy to move back and forth.  And, if you have a team familiar (having spent months or years on this project) with your situation that is best suited to fix things, they may only have the ability to focus on one system at a time.  I have read they moved to this system due to the old one aging out.  That may also make it hard to go back if support for the old system has faded away.
 
Essentially, once you commit to going live, it isn't easy turning back the clock, if you can do it all.  It may be better to put your limited resources into fixing the new system pronto.

As to the CEO being on the board of Nymbus, I don't think it is a conflict because I read Vystar actually owns a good sized chunk of Nymbus.  Thus, as a key investor, it would be appropriate for them to be represented on its board.  What is a double whammy is not only does Vystar take a PR or more hit for this issue, but it also may substantially devalue their investment in Nymbus, at least in the short run.

Depends on how it's architected. Clearly their digital platform is not their core banking system, or this would have truly been catastrophic. That means that the data appears online either through a real-time call, or through some sort of sync process but either way the core system is the "system of record", so you could POSSIBLY have both up at one time. Now, without knowing how their systems work behind the scenes there's no way to know if that was realistic.

[Steve]

So, as an update - they have a queueing thing on their site now, which required me to wait about 10-15 minutes before giving me a login prompt. Once I logged in the site performed fine. Not the fastest thing I've ever seen but completely acceptable speed. Additionally, it appears the balance and transactions are accurate.

BUT, all of the transaction descriptions are trash. Here is one of mine:

"DBT 14:39 5/23/22 005539 PURCHASE PIN, YES"

Literally that's it. The descriptions weren't great in the old system and this was something that was supposed to be a benefit of the new system. When I could get in immediately after cutover the descriptions were really nice and clear (and had some soft touches like the merchant's logo, etc.), so I'm wondering if whatever the system was doing to translate the descriptions was creating a lot of load?

Also, there are no transactions displayed for anything but the Checking account. I mean between checking, savings, and my car loan the checking is certainly the most volatile where I need the transactions the most, but yikes.

There were clearly aspects of this system that were not tested well at all.

[pierre]

I was able to get in through the website after a 20 minute or so wait in queue. After about 2 minutes in, the site locked up and I had to try again. It’s probably the least they can do for now. But still not a great look for VyStar.