Jump to content

Miscellaneous TTC Discussion & Questions


Orion V
 Share

Recommended Posts

41 minutes ago, MK78 said:

I'm wondering what are the chances that they can have VISION/NEXTBUS working for Monday morning service... 

That would be nice - but I'd think the biggest issue is that Wheel-Trans can't book rides.

I'm surprised (no .. I'm not really) that there isn't a service alert out there for the Wheel-Trans and Nextbus issues.

I don't see how the TTC CIO doesn't get fired after this, when other agencies were already hit. Bad enough that they weren't good enough to stop something getting in. How wasn't the entire thing not resetable by going to a backup overnight?
 

 

Link to comment
Share on other sites

I'm hearing some dispatch to bus transmission on the normal "channels" like is usually the case on VISION here in the east end, but the fallback channel is still being used, so maybe it's slowly being brought back.

But yeah, the Wheel-Trans situation is not pretty if people can't book it.

 

Edit: nevermind, I just heard it's still down. and those transmissions have stopped, so maybe they are trying to cobble something together.

Link to comment
Share on other sites

5 hours ago, sdgta2008 said:

First STM, then STO, now the TTC....  I'm not liking this pattern.

 

2 hours ago, nfitz said:

The pattern seems to be alphabetical - but are they spelling it out, or just using the abbreviation. If the former, than Translink should watch out! If not, then Victoria is next.

By the same logic they could be heading westbound lol

  • Like 1
  • Haha 1
Link to comment
Share on other sites

18 hours ago, MK78 said:

I'm hearing some dispatch to bus transmission on the normal "channels" like is usually the case on VISION here in the east end, but the fallback channel is still being used, so maybe it's slowly being brought back.

But yeah, the Wheel-Trans situation is not pretty if people can't book it.

 

Edit: nevermind, I just heard it's still down. and those transmissions have stopped, so maybe they are trying to cobble something together.

I guess they don't have a backup system?

Link to comment
Share on other sites

Well I hope heads will roll for whoever implemented and "secured" the system. I still wonder if it had anything to do with the TTC site being redone just days earlier, and a vulnerability was exploited to get deep into the core.

  • Like 2
Link to comment
Share on other sites

19 hours ago, MK78 said:

Well I hope heads will roll for whoever implemented and "secured" the system. I still wonder if it had anything to do with the TTC site being redone just days earlier, and a vulnerability was exploited to get deep into the core.

No.

Link to comment
Share on other sites

22 hours ago, MK78 said:

Well I hope heads will roll for whoever implemented and "secured" the system. I still wonder if it had anything to do with the TTC site being redone just days earlier, and a vulnerability was exploited to get deep into the core.

From what I gather,

Not in this case, but not impossible in other scenarios. The TTC sets up their servers a bit differently from other local agencies; they keep their website servers and general comms servers separate. The website is in what's called a DMZ, ie. a highly protected (more beefy than normal) seperate server that has no links to the rest of the TTC's network, not even physically. If you noticed, the TTC website never really went down, as where VISION, tracking, general comms went down, as they're in seperate servers. The DMZ had to be opened and closed to patch the new website, not the general servers, meaning that it was a coincidence. 
However, if someone inside the TTC wanted to wreak havoc on the DMZ server AND the other servers at once, that is possible, but one cannot cause the other. 

Link to comment
Share on other sites

On 10/30/2021 at 5:35 PM, nfitz said:

 don't see how the TTC CIO doesn't get fired after this, when other agencies were already hit. Bad enough that they weren't good enough to stop something getting in. How wasn't the entire thing not resetable by going to a backup overnight?

Probably because it isn't that simple. The TTC isn't just a bunch of servers; I imagine there are thousands of desktop PCs and laptops on the network as well. All it takes is one of those machines to still be compromised, and you're back to being hacked.

I imagine IT has literally pulled the plug on the network to ensure that everything is isolated from one another, and are slowly going through and checking each device one by one before even thinking about reconnecting. Plus, law enforcement is going to be involved, and are going to want forensic data to comb through, so you can't even just wipe-and-reload the servers with the backup because that'll destroy the data they want preserved.

And all that assumes your backups aren't compromised to begin with...

  • Like 1
Link to comment
Share on other sites

That ransomware attack thing is ridiculous.. Nextbus app is crucial especially when your bus route isn’t that frequent (like 20-30 min per bus).

The schedule is kinda useless since the bus doesn’t even come according to it.. and there’s no fxxxin way that I can see when the bus is coming.. 

I waited more than 10 minute past the scheduled time and forced to take an uber to subway.

man.. I think the TTC could lose a bit of revenue because of this

Link to comment
Share on other sites

Just noticed the Queen St section between Yonge to University track and overhead wires were just upgraded which begs this question: Aren't they planning to close this section down in 2 years time for a 4.5 years construction on the OL? If so, wouldn't they rip this stuff all out or can those constructing machines avoid the tracks and overhead catenary?

And yes I know the streetcar upgrade was planned well in advance while the OL closure was just announced a few months ago. Couldn't they have coordinated it better and left the upgrade alone since streetcar service will only run for 2 more years before the lengthy closure for the OL construction.

Link to comment
Share on other sites

1 hour ago, Orion VI said:

From what I gather,

Not in this case, but not impossible in other scenarios. The TTC sets up their servers a bit differently from other local agencies; they keep their website servers and general comms servers separate. The website is in what's called a DMZ, ie. a highly protected (more beefy than normal) seperate server that has no links to the rest of the TTC's network, not even physically. If you noticed, the TTC website never really went down, as where VISION, tracking, general comms went down, as they're in seperate servers. The DMZ had to be opened and closed to patch the new website, not the general servers, meaning that it was a coincidence. 
However, if someone inside the TTC wanted to wreak havoc on the DMZ server AND the other servers at once, that is possible, but one cannot cause the other. 

How does the NextBus data get sent to customer-facing websites & apps? It has to be connected to the system somehow.

1 hour ago, raptorjays said:

That ransomware attack thing is ridiculous.. Nextbus app is crucial especially when your bus route isn’t that frequent (like 20-30 min per bus).

Indeed it's a crapshoot, my morning bus of choice comes every 25-30 minutes, and it varies on the operator how fast they drive, so the tracking is important to me as well. Lucky this morning he came on time.

I could take other routes if I know he's not coming (it has happened where that particular run number hasnt left the garage or it's very late, as it's a deadhead run to the top of the route), but I like it because its not crowded at all.

But yeah, I've very much gotten used to the live tracking. It's been around for what, over 10 years now?

Link to comment
Share on other sites

1 hour ago, Archer said:

Probably because it isn't that simple. The TTC isn't just a bunch of servers; I imagine there are thousands of desktop PCs and laptops on the network as well. All it takes is one of those machines to still be compromised, and you're back to being hacked.

I imagine IT has literally pulled the plug on the network to ensure that everything is isolated from one another, and are slowly going through and checking each device one by one before even thinking about reconnecting. Plus, law enforcement is going to be involved, and are going to want forensic data to comb through, so you can't even just wipe-and-reload the servers with the backup because that'll destroy the data they want preserved.

And all that assumes your backups aren't compromised to begin with...

It actually is this simple, just not in this case. As I mentioned earlier, the website didn't get hit, so the backup @nfitz references would be one for VISION and mywheeltrans booking (which is completley different than a website backup). If ONLY the website was affected, major agencies will have a backup that is relatively up to date with the front-facing one. They might lose updates that happened within recent time to the website (ie. specific closures and advisories, or job postings), but everything else will be fine. 

As for IT pulling the plug, yes, but they wouldn't comb each PC, it would be more along the lines of testing servers in general to try and spot and abnormal files, or things that shouldn't be there. Unless they had specific data pointing to the fact that this was from a link that shouldn't have been clicked (ie. email or chat servers, which in fairness could very well be the case), then combing would just be a waste of time, especially if the computer's data is stored on their own servers. 

In terms of the Police getting involved: probably not. The authorities typically get involved when it is a matter of national security, ie. if this happened to Ottawa (who has VERY good protection, for this specific reason). A third party investigation company might be involved, but at the expense of the Municipal government. For example, in New Jersey, (NJ Transit) stuff like this will get mentioned in government meetings, but police aren't typically playing a big role. Take the Translink example as a reference: Police weren't involved, only Transit Police and outside cybersecurity experts according to some articles. (THey mention the RCMP investigating multiple attacks, but not specifically that one). https://globalnews.ca/news/7499986/translink-suspicious-network-activity-update/ https://bc.ctvnews.ca/printed-ransom-note-asked-translink-for-7-5-million-in-december-cyberattack-1.5389170

Often the response to an event like this is to just rebuild the servers from scratch, with the companies that built them in the first place. 

1 hour ago, MK78 said:

How does the NextBus data get sent to customer-facing websites & apps? It has to be connected to the system somehow.

No. Notice how the website doesn't track live updates, and only third-party apps and in-station displays do? The website server isn't the main wireless communication system that the TTC runs everything through. The apps get live updates from a seperate server, and turn it into readable nextbus info.

Link to comment
Share on other sites

20 hours ago, Orion VI said:

From what I gather,

Not in this case, but not impossible in other scenarios. The TTC sets up their servers a bit differently from other local agencies; they keep their website servers and general comms servers separate. The website is in what's called a DMZ, ie. a highly protected (more beefy than normal) seperate server that has no links to the rest of the TTC's network, not even physically. If you noticed, the TTC website never really went down, as where VISION, tracking, general comms went down, as they're in seperate servers. The DMZ had to be opened and closed to patch the new website, not the general servers, meaning that it was a coincidence. 
However, if someone inside the TTC wanted to wreak havoc on the DMZ server AND the other servers at once, that is possible, but one cannot cause the other. 

Website didn't go down, but TTC has lost the ability to update it - according to customer service.
 

 

Link to comment
Share on other sites

I was on 8964 on the 12C yesterday and I noticed that the entire NSS display was turned off and not functioning at all, I also noticed that there was a radio next to the operator, which you could constantly hear other operators and TC talking through. 

Link to comment
Share on other sites

2 hours ago, blue.bird.fan said:

It looks like tracking is back up and running.

Sweet. Was a strange few days waiting for buses not knowing where they are. We've been spoiled by this tracking feature for 10+ years now.

Link to comment
Share on other sites

1 hour ago, Archer said:

Funny that was their priority, since they haven't bothered to try and fix the pay system to ensure that everyone gets paid properly. 🙄

Sounds like they know exactly where the average Torontonian's priorities lie.

Who cares about the frontline workers who provide them the service? As long as the customers get what they want, exactly when they demand it. :rolleyes:

  • Haha 2
Link to comment
Share on other sites

1 hour ago, MK78 said:

Sweet. Was a strange few days waiting for buses not knowing where they are. We've been spoiled by this tracking feature for 10+ years now.

I did, however, get to realize how transit fans looked for specific buses before NextBus was a thing, so that was fun.

  • Like 1
Link to comment
Share on other sites

12 hours ago, PCC Guy said:

Sounds like they know exactly where the average Torontonian's priorities lie.

Who cares about the frontline workers who provide them the service? As long as the customers get what they want, exactly when they demand it. :rolleyes:

lip approves of this message.

  • Like 1
Link to comment
Share on other sites

13 hours ago, PCC Guy said:

Sounds like they know exactly where the average Torontonian's priorities lie.

Who cares about the frontline workers who provide them the service? As long as the customers get what they want, exactly when they demand it. :rolleyes:

Oh for sure.  The front line workers are there to be screamed at when dissatisfaction needs to be expressed and the back line staff are the nasty little urchins tasked with making sure everything still happens no matter what.  If they have to sit without pay for a few weeks, then so be it, as long as the customer facing conveniences are gotten up and running pronto.  I actually kind of suspect that might have had something to do with it.  The TTC very publicly announced in all the media that they were restoring systems and by getting the tracking etc. running again, they could go back out and say see, we're getting stuff running again, tracking's now available while someone quietly tells the employees to make arrangements with their landlords and lending institutions to get extension on the October end of month bills and rent/mortgage payments because payroll's still down.

To be fair to the TTC though, if the tracking system was new enough to be done in virtual machines running on some big iron, that could have legitmately a fast recovery by rolling back through VM snapshots and firing up the most recent one before ransomeware infection.  If the payroll system is older, this most likely isn't possible, and putting it back together or blowing away the disks and restoring backups including all the data could be painful - especially if there is an investigation going on that needs the equipment to be held.  That's not as straightforward as sequestering the virtual machine and spinning up a new replacement.

SEPTA went through a similar ransomeware cyberattack over the last year and the FBI added it to their ongoing investigation of transit authority hacks and they had serious, significant months of downtime.  The somewhat secret scandal was with COVID-19, all of the 1234 Market St. managerial types were "working from home" but had nothing to connect to on the back end which severely limited the amount of actual work that could be done so it turned into a big paid vacation for the office workers.  Meanwhile, the operators kept service running, and the maintenance side of things reverted to an old school paper and whiteboard way of administrating everything.  I think, and I'd have to ask to double check, that some of the delay in getting stuff back there was because of the FBI investigation and the need to hold the vandalized equipment meaning their IT people couldn't blow servers away and reinstall the OS and application software.

12 hours ago, blue.bird.fan said:

I did, however, get to realize how transit fans looked for specific buses before NextBus was a thing, so that was fun.

Cool - since that's a new experience for you, what are your thoughts on it?

We should do a couple more throwbacks, we'll have to introduce you to driving a car with not GPS or cellphone, programming a VCR, and how to manage a living room with an AM/FM stereo receiver, turntable and cassete deck and a TV set that all have have no remote controls.

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...