Archer Posted November 1, 2021 Report Share Posted November 1, 2021 On 10/30/2021 at 5:35 PM, nfitz said: don't see how the TTC CIO doesn't get fired after this, when other agencies were already hit. Bad enough that they weren't good enough to stop something getting in. How wasn't the entire thing not resetable by going to a backup overnight? Probably because it isn't that simple. The TTC isn't just a bunch of servers; I imagine there are thousands of desktop PCs and laptops on the network as well. All it takes is one of those machines to still be compromised, and you're back to being hacked. I imagine IT has literally pulled the plug on the network to ensure that everything is isolated from one another, and are slowly going through and checking each device one by one before even thinking about reconnecting. Plus, law enforcement is going to be involved, and are going to want forensic data to comb through, so you can't even just wipe-and-reload the servers with the backup because that'll destroy the data they want preserved. And all that assumes your backups aren't compromised to begin with... 1 Link to comment Share on other sites More sharing options...
raptorjays Posted November 1, 2021 Report Share Posted November 1, 2021 That ransomware attack thing is ridiculous.. Nextbus app is crucial especially when your bus route isn’t that frequent (like 20-30 min per bus). The schedule is kinda useless since the bus doesn’t even come according to it.. and there’s no fxxxin way that I can see when the bus is coming.. I waited more than 10 minute past the scheduled time and forced to take an uber to subway. man.. I think the TTC could lose a bit of revenue because of this Link to comment Share on other sites More sharing options...
Orion V Posted November 2, 2021 Author Report Share Posted November 2, 2021 Just noticed the Queen St section between Yonge to University track and overhead wires were just upgraded which begs this question: Aren't they planning to close this section down in 2 years time for a 4.5 years construction on the OL? If so, wouldn't they rip this stuff all out or can those constructing machines avoid the tracks and overhead catenary? And yes I know the streetcar upgrade was planned well in advance while the OL closure was just announced a few months ago. Couldn't they have coordinated it better and left the upgrade alone since streetcar service will only run for 2 more years before the lengthy closure for the OL construction. Link to comment Share on other sites More sharing options...
MK78 Posted November 2, 2021 Report Share Posted November 2, 2021 1 hour ago, Orion VI said: From what I gather, Not in this case, but not impossible in other scenarios. The TTC sets up their servers a bit differently from other local agencies; they keep their website servers and general comms servers separate. The website is in what's called a DMZ, ie. a highly protected (more beefy than normal) seperate server that has no links to the rest of the TTC's network, not even physically. If you noticed, the TTC website never really went down, as where VISION, tracking, general comms went down, as they're in seperate servers. The DMZ had to be opened and closed to patch the new website, not the general servers, meaning that it was a coincidence. However, if someone inside the TTC wanted to wreak havoc on the DMZ server AND the other servers at once, that is possible, but one cannot cause the other. How does the NextBus data get sent to customer-facing websites & apps? It has to be connected to the system somehow. 1 hour ago, raptorjays said: That ransomware attack thing is ridiculous.. Nextbus app is crucial especially when your bus route isn’t that frequent (like 20-30 min per bus). Indeed it's a crapshoot, my morning bus of choice comes every 25-30 minutes, and it varies on the operator how fast they drive, so the tracking is important to me as well. Lucky this morning he came on time. I could take other routes if I know he's not coming (it has happened where that particular run number hasnt left the garage or it's very late, as it's a deadhead run to the top of the route), but I like it because its not crowded at all. But yeah, I've very much gotten used to the live tracking. It's been around for what, over 10 years now? Link to comment Share on other sites More sharing options...
Orion VI Posted November 2, 2021 Report Share Posted November 2, 2021 1 hour ago, Archer said: Probably because it isn't that simple. The TTC isn't just a bunch of servers; I imagine there are thousands of desktop PCs and laptops on the network as well. All it takes is one of those machines to still be compromised, and you're back to being hacked. I imagine IT has literally pulled the plug on the network to ensure that everything is isolated from one another, and are slowly going through and checking each device one by one before even thinking about reconnecting. Plus, law enforcement is going to be involved, and are going to want forensic data to comb through, so you can't even just wipe-and-reload the servers with the backup because that'll destroy the data they want preserved. And all that assumes your backups aren't compromised to begin with... It actually is this simple, just not in this case. As I mentioned earlier, the website didn't get hit, so the backup @nfitz references would be one for VISION and mywheeltrans booking (which is completley different than a website backup). If ONLY the website was affected, major agencies will have a backup that is relatively up to date with the front-facing one. They might lose updates that happened within recent time to the website (ie. specific closures and advisories, or job postings), but everything else will be fine. As for IT pulling the plug, yes, but they wouldn't comb each PC, it would be more along the lines of testing servers in general to try and spot and abnormal files, or things that shouldn't be there. Unless they had specific data pointing to the fact that this was from a link that shouldn't have been clicked (ie. email or chat servers, which in fairness could very well be the case), then combing would just be a waste of time, especially if the computer's data is stored on their own servers. In terms of the Police getting involved: probably not. The authorities typically get involved when it is a matter of national security, ie. if this happened to Ottawa (who has VERY good protection, for this specific reason). A third party investigation company might be involved, but at the expense of the Municipal government. For example, in New Jersey, (NJ Transit) stuff like this will get mentioned in government meetings, but police aren't typically playing a big role. Take the Translink example as a reference: Police weren't involved, only Transit Police and outside cybersecurity experts according to some articles. (THey mention the RCMP investigating multiple attacks, but not specifically that one). https://globalnews.ca/news/7499986/translink-suspicious-network-activity-update/ https://bc.ctvnews.ca/printed-ransom-note-asked-translink-for-7-5-million-in-december-cyberattack-1.5389170 Often the response to an event like this is to just rebuild the servers from scratch, with the companies that built them in the first place. 1 hour ago, MK78 said: How does the NextBus data get sent to customer-facing websites & apps? It has to be connected to the system somehow. No. Notice how the website doesn't track live updates, and only third-party apps and in-station displays do? The website server isn't the main wireless communication system that the TTC runs everything through. The apps get live updates from a seperate server, and turn it into readable nextbus info. Link to comment Share on other sites More sharing options...
nfitz Posted November 2, 2021 Report Share Posted November 2, 2021 20 hours ago, Orion VI said: From what I gather, Not in this case, but not impossible in other scenarios. The TTC sets up their servers a bit differently from other local agencies; they keep their website servers and general comms servers separate. The website is in what's called a DMZ, ie. a highly protected (more beefy than normal) seperate server that has no links to the rest of the TTC's network, not even physically. If you noticed, the TTC website never really went down, as where VISION, tracking, general comms went down, as they're in seperate servers. The DMZ had to be opened and closed to patch the new website, not the general servers, meaning that it was a coincidence. However, if someone inside the TTC wanted to wreak havoc on the DMZ server AND the other servers at once, that is possible, but one cannot cause the other. Website didn't go down, but TTC has lost the ability to update it - according to customer service. Link to comment Share on other sites More sharing options...
TTC7957 Posted November 3, 2021 Report Share Posted November 3, 2021 I was on 8964 on the 12C yesterday and I noticed that the entire NSS display was turned off and not functioning at all, I also noticed that there was a radio next to the operator, which you could constantly hear other operators and TC talking through. Link to comment Share on other sites More sharing options...
blue.bird.fan Posted November 3, 2021 Report Share Posted November 3, 2021 It looks like tracking is back up and running. 2 Link to comment Share on other sites More sharing options...
Archer Posted November 4, 2021 Report Share Posted November 4, 2021 49 minutes ago, blue.bird.fan said: It looks like tracking is back up and running. Funny that was their priority, since they haven't bothered to try and fix the pay system to ensure that everyone gets paid properly. 1 Link to comment Share on other sites More sharing options...
MK78 Posted November 4, 2021 Report Share Posted November 4, 2021 2 hours ago, blue.bird.fan said: It looks like tracking is back up and running. Sweet. Was a strange few days waiting for buses not knowing where they are. We've been spoiled by this tracking feature for 10+ years now. Link to comment Share on other sites More sharing options...
T3G Posted November 4, 2021 Report Share Posted November 4, 2021 1 hour ago, Archer said: Funny that was their priority, since they haven't bothered to try and fix the pay system to ensure that everyone gets paid properly. Sounds like they know exactly where the average Torontonian's priorities lie. Who cares about the frontline workers who provide them the service? As long as the customers get what they want, exactly when they demand it. 2 Link to comment Share on other sites More sharing options...
blue.bird.fan Posted November 4, 2021 Report Share Posted November 4, 2021 1 hour ago, MK78 said: Sweet. Was a strange few days waiting for buses not knowing where they are. We've been spoiled by this tracking feature for 10+ years now. I did, however, get to realize how transit fans looked for specific buses before NextBus was a thing, so that was fun. 1 Link to comment Share on other sites More sharing options...
Bus_Medic Posted November 4, 2021 Report Share Posted November 4, 2021 12 hours ago, PCC Guy said: Sounds like they know exactly where the average Torontonian's priorities lie. Who cares about the frontline workers who provide them the service? As long as the customers get what they want, exactly when they demand it. lip approves of this message. 1 Link to comment Share on other sites More sharing options...
Wayside Observer Posted November 4, 2021 Report Share Posted November 4, 2021 13 hours ago, PCC Guy said: Sounds like they know exactly where the average Torontonian's priorities lie. Who cares about the frontline workers who provide them the service? As long as the customers get what they want, exactly when they demand it. Oh for sure. The front line workers are there to be screamed at when dissatisfaction needs to be expressed and the back line staff are the nasty little urchins tasked with making sure everything still happens no matter what. If they have to sit without pay for a few weeks, then so be it, as long as the customer facing conveniences are gotten up and running pronto. I actually kind of suspect that might have had something to do with it. The TTC very publicly announced in all the media that they were restoring systems and by getting the tracking etc. running again, they could go back out and say see, we're getting stuff running again, tracking's now available while someone quietly tells the employees to make arrangements with their landlords and lending institutions to get extension on the October end of month bills and rent/mortgage payments because payroll's still down. To be fair to the TTC though, if the tracking system was new enough to be done in virtual machines running on some big iron, that could have legitmately a fast recovery by rolling back through VM snapshots and firing up the most recent one before ransomeware infection. If the payroll system is older, this most likely isn't possible, and putting it back together or blowing away the disks and restoring backups including all the data could be painful - especially if there is an investigation going on that needs the equipment to be held. That's not as straightforward as sequestering the virtual machine and spinning up a new replacement. SEPTA went through a similar ransomeware cyberattack over the last year and the FBI added it to their ongoing investigation of transit authority hacks and they had serious, significant months of downtime. The somewhat secret scandal was with COVID-19, all of the 1234 Market St. managerial types were "working from home" but had nothing to connect to on the back end which severely limited the amount of actual work that could be done so it turned into a big paid vacation for the office workers. Meanwhile, the operators kept service running, and the maintenance side of things reverted to an old school paper and whiteboard way of administrating everything. I think, and I'd have to ask to double check, that some of the delay in getting stuff back there was because of the FBI investigation and the need to hold the vandalized equipment meaning their IT people couldn't blow servers away and reinstall the OS and application software. 12 hours ago, blue.bird.fan said: I did, however, get to realize how transit fans looked for specific buses before NextBus was a thing, so that was fun. Cool - since that's a new experience for you, what are your thoughts on it? We should do a couple more throwbacks, we'll have to introduce you to driving a car with not GPS or cellphone, programming a VCR, and how to manage a living room with an AM/FM stereo receiver, turntable and cassete deck and a TV set that all have have no remote controls. 1 Link to comment Share on other sites More sharing options...
IRT_BMT_IND Posted November 4, 2021 Report Share Posted November 4, 2021 14 hours ago, blue.bird.fan said: I did, however, get to realize how transit fans looked for specific buses before NextBus was a thing, so that was fun. The TTC actually had live vehicle tracking as part of the old CIS system (I think it first went live in the early 80s and was rolled out on a per-division basis, someone here probably knows more details) through an IVR telephone system (and I think fax too). This was actually cutting edge technology for the time (A lot of it came from Bell and I think either Nortel or companies in Nortel's orbit) and the TTC was one of the first transit agencies in the world to have anything like this. It was shut down in the late 90s for Y2K compliance reasons and not replaced until the nextbus system went online in the 00s. Link to comment Share on other sites More sharing options...
Wayside Observer Posted November 4, 2021 Report Share Posted November 4, 2021 47 minutes ago, IRT_BMT_IND said: The TTC actually had live vehicle tracking as part of the old CIS system (I think it first went live in the early 80s and was rolled out on a per-division basis, someone here probably knows more details) through an IVR telephone system (and I think fax too). This was actually cutting edge technology for the time (A lot of it came from Bell and I think either Nortel or companies in Nortel's orbit) and the TTC was one of the first transit agencies in the world to have anything like this. It was shut down in the late 90s for Y2K compliance reasons and not replaced until the nextbus system went online in the 00s. You're confusing two different things. CIS brought out vehicle locations to the divisions and transit control on a bar chart type readout showing where it was relative to the length of the route. None of that was publicly accessible. Timeline, the phone system was very impressive for the late 80s but all it told you was arrival times for the next couple of buses coming to your stop. It didn't let you find out where a given bus was in real time or even give revised arrival times accounting for delays or current vehicle position in CIS; the two were separate. I don't know if they had any Y2K problems with the CIS infrastructure that had to be patched but Timeline definitely did and that's what put it out of business. But while it was running, nobody was running to a payphone to call in and find out where in Toronto their favourite bus was running because it didn't have that kind of equivalent functionality to what there is on the various websites/apps that pull near realtime data these days. Link to comment Share on other sites More sharing options...
Bus_Medic Posted November 4, 2021 Report Share Posted November 4, 2021 54 minutes ago, IRT_BMT_IND said: The TTC actually had live vehicle tracking as part of the old CIS system (I think it first went live in the early 80s and was rolled out on a per-division basis, someone here probably knows more details) through an IVR telephone system (and I think fax too). This was actually cutting edge technology for the time (A lot of it came from Bell and I think either Nortel or companies in Nortel's orbit) and the TTC was one of the first transit agencies in the world to have anything like this. It was shut down in the late 90s for Y2K compliance reasons and not replaced until the nextbus system went online in the 00s. I remember having to replace the wheel rotation sensors on the right front drum. The air gap was really finicky. The microwave beacons, UHF radio and brake drum pick ups were in use until the CDMA mod was finished up in ‘07 or ‘08. Link to comment Share on other sites More sharing options...
Wayside Observer Posted November 4, 2021 Report Share Posted November 4, 2021 Oh yeah, I forgot the fax-back system. You could call into that number with a touch-tone phone, dial through the menu to request a schedule, input your fax line's number, and it would fax you the timetable you requested. Or, and I know someone who did this, input someone else's fax line number and order a ton of schedules burning their paper and tying their line up. That was one step technologically above getting the subscription cards out of magazines in the news store, signing someone you didn't like up for a bunch, and ticking the "Bill me later" boxes on the cards. But yeah, fax-backs were a thing once upon a time but hardly anyone had a fax machine or a fax modem with a computer on all the time to answer it at home so it was more of a buisness-to-business thing. If you were at work, you could send your TTC schedule to the office fax machine and pick it up there. Lots of companies had fax-back systems where you could go through a touch-tone menu and request catalogs, information sheets, technical service bulletins and get them sent to the nearest fax machine back in the day. Good grief, if I get any older, they're going to stuff and mount me and put me on display at HCRR. 3 Link to comment Share on other sites More sharing options...
Xtrazsteve Posted November 5, 2021 Report Share Posted November 5, 2021 9 hours ago, Wayside Observer said: Oh yeah, I forgot the fax-back system. You could call into that number with a touch-tone phone, dial through the menu to request a schedule, input your fax line's number, and it would fax you the timetable you requested. Or, and I know someone who did this, input someone else's fax line number and order a ton of schedules burning their paper and tying their line up. That was one step technologically above getting the subscription cards out of magazines in the news store, signing someone you didn't like up for a bunch, and ticking the "Bill me later" boxes on the cards. But yeah, fax-backs were a thing once upon a time but hardly anyone had a fax machine or a fax modem with a computer on all the time to answer it at home so it was more of a buisness-to-business thing. If you were at work, you could send your TTC schedule to the office fax machine and pick it up there. Lots of companies had fax-back systems where you could go through a touch-tone menu and request catalogs, information sheets, technical service bulletins and get them sent to the nearest fax machine back in the day. Good grief, if I get any older, they're going to stuff and mount me and put me on display at HCRR. Ahhh. I actually remember requesting some route info through that system. That was doable the late 90s. Although the quality of my fax machine wasn't that great and it ended up either very black or white. Link to comment Share on other sites More sharing options...
Wayside Observer Posted November 7, 2021 Report Share Posted November 7, 2021 Talking on the phone with a friend and tossing around some potential ideas for 2022 and the subject of charters came up. Does the TTC still have the panedmic related ban on charters? Does it look like the pandemic ban on charters is going to become permanent, ie. become the final exit from the charter business that the TTC's wanted to get out of for a long time? Link to comment Share on other sites More sharing options...
raptorjays Posted November 8, 2021 Report Share Posted November 8, 2021 Anyone know why the next train display isn’t working?? it is only saying “every 3 min” “every 5 min” rather than the actual time Link to comment Share on other sites More sharing options...
MK78 Posted November 8, 2021 Report Share Posted November 8, 2021 9 minutes ago, raptorjays said: Anyone know why the next train display isn’t working?? it is only saying “every 3 min” “every 5 min” rather than the actual time Most likely still related to the ransomware attack, some parts of the system may not be restored yet. 2 Link to comment Share on other sites More sharing options...
Orion VI Posted November 9, 2021 Report Share Posted November 9, 2021 On 11/8/2021 at 1:35 PM, MK78 said: Most likely still related to the ransomware attack, some parts of the system may not be restored yet. I think he's asking because the nextbus times have been working for a couple of days, but the "nextsubway" times haven't been. Even Transsee and related apps can track and predict subway arrival times as of now, but somehow the subway online system doesn't. Link to comment Share on other sites More sharing options...
smallspy Posted November 10, 2021 Report Share Posted November 10, 2021 17 hours ago, Orion VI said: I think he's asking because the nextbus times have been working for a couple of days, but the "nextsubway" times haven't been. Even Transsee and related apps can track and predict subway arrival times as of now, but somehow the subway online system doesn't. The subway system is completely independent from VISION and so its prediction system uses different computers that interface with its signaling systems. It's quite likely that it just hasn't been brought back online yet. After all, apparently the internal email system is still down as of last night. Dan 3 1 Link to comment Share on other sites More sharing options...
John Oke Posted November 10, 2021 Report Share Posted November 10, 2021 https://www.blogto.com/city/2021/11/ttc-phone-sex-hotline/ if this was intentional, someone is probably getting fired for this. 1 Link to comment Share on other sites More sharing options...
Recommended Posts
Please sign in to comment
You will be able to leave a comment after signing in
Sign In Now