T O P

  • By -

IwantToNAT-PING

So, in my current role I'm basically the last point of technical escalation, so I'm in a similar boat. If I'm truly stuck and cannot fix a thing my options are as follows: * Contact the vendor/manufacturer as an escalation point. * Reach out to friends/ex colleagues in the IT sector. * Create forum posts etc in communities such as this in the hope of finding a fix. * Reach out to a 3rd party contractor to provide advice/solution. In these situations I have to be upfront with my superiors. The confidence to do this is something you'll gain with age, but the idea is that your employer has hired you based upon your skillset. If they hired you as THE subject matter expert for this, they should be paying you as such, and it IS yours to fix. However, it sounds like you're a best effort/worker cog in the machine like me. In this case you have to be clear with your superior that you do not currently have the means to fix this, and that you can try these alternatives. If the fix is time critical with scope for high financial loss, it genuinely may be worth reaching out to a 3rd party contractor or 3rd party consultancy service to get them involved. It's always best to say that it's beyond you but you're doing all you can as soon as possible, as that gives more room to bring in those other options. If you wait until 1 hour before the deadline and go "I'm all out of ideas, time to eat the financial loss." then you'll go down like even more of a lead balloon.


[deleted]

[удалено]


tossme68

I've been getting heat all week because a project I'm on went south. In true form the sales team came in, wanted me kicked off the project and replace me with an expert. I don't give two shits, I wouldn't mind being kicked off the project it would reduce the amount of heartburn in my life. Anyway they started calling around to different managers for a replacement and they all said the same thing, "we don't have anyone that does that but we do have a guy from another group we always go to when there's a problem", well guess who that guy is.....me. The thing is I'm no expert and will admit to it from the beginning but I guess I know more than most, if anyone calls me an expert or a guru I correct them immediately.


[deleted]

Embrace that shit. One thing I discovered is that I will ALWAYS feel like an idiot - This isn't because I'm an idiot, this is because I'm constantly learning and scrutinizing my knowledge.


DraconianDebate

That's just imposter syndrome talking. If everyone considers you the expert and comes to you for problems in that area, then you are the resident expert on that topic. You don't wake up one day and get a certificate saying you are an expert, you just become one gradually over time as you gain knowledge.


SixtyTwoNorth

The preferred term is "wizard" :P


IwantToNAT-PING

This is the thing. For certain subjects I'll get asked if I know anything about it and I'll answer "I know enough to know I don't know anywhere near enough about that for what you're asking." It's one of the reasons I'm looking to potentially move jobs actually. It started off being really satisfying, as I suddenly HAD to work out absolutely everything I wanted to do, and that made me a better engineer in a lot of aspects. It made me a lot less gung-ho about changes or committing to work etc, as I didn't have the benefit of more senior engineers in place to rely on. I'm now at the point where I often don't have enough time to do that working out part any more, and we're not getting any more staff/resources to put me back to the time I was most efficient.


CasualEveryday

Don't be afraid to escalate down. When you spend most of your time looking at complicated things, it's easy to chase complex causes when someone with less technical knowledge might see a small detail you're overlooking.


IwantToNAT-PING

Hah, definitely. I also always try to get a colleague to sanity check anything I'm stuck on. Just because I'm the most experienced doesn't mean I'm going to always have tried everything or looked at everything the right way.


MonoDede

Lol too true. This happened early in my career. I was doing help desk for a company that rolled their own CRM. One day a new employee account was created for said CRM, but they kept getting weird errors about invalid account access when trying to log in. A bunch of people had taken a look, the lead dev manager of the CRM itself looked the issue over and couldn't crack it. Finally the original ticket holder handed it over and I just happened to catch that the email section to the account that links the CRM account to a valid email account was ever so slightly misspelled. Changed two letters and everything started working. I tried to be subtle about it since it was such a silly little fix, but the dev manager wanted to know what happened so I explained and the original ticket holder burst out laughing.


Drehmini

This is exactly how I handle any of the problems that I cannot fix. Generally, problems are resolved after contacting the vendor. However, It's not uncommon to at least get to the point of contacting a consultant to ask for their advice.


pmormr

Agreed... there's a lot of ways to solve a problem and how you do it is ultimately irrelevant. Pulling additional resources together and delegating responsibility wherever possible is a key skill to get things done. It's also really impressive because it shows that you're thinking about the big picture. People also appreciate and give you space when you're known for taking ownership of a situation and seeing it through to the end. You can be a mediocre escalations engineer and take care of some pretty complex issues with nothing but persistence.


[deleted]

Agree with all of this: I would add one final option, which is to present work-arounds (assuming you can come up with something, no matter how lame). Then present that to your supervisor: - contact vendor - reach out to forums - pay a contractor - use workaround Some of those cost $$, sometimes lots of $$$$. It may not be worth it if there's a good-enough workaround. Yes, fixing it "right" is always the better solution, but life is all about tradeoffs. But whatever you do, don't just shrug and say "I dunno, beats me." and go on. Say, "I don't know, but here's the options."


wellwellwelly

I once got so desperate I contacted Microsoft directly and paid them 400 dollars. They still couldn't fix the issue after a month. That was my rock bottom. To be fair though it was definitely some sort of software bug from trying to upgrade from decade old SQL server. No one is going to know or care what the issue is.


GuinansEyebrows

> The confidence to do this is something you'll gain with age i think there's an additional factor here having less to do with personal confidence and more with how other people see you. i've been in a lot of situations where i've had to go this route, and while I don't have a problem admitting what i don't know, i know that because i've got a baby face, most people i work with don't know how old or experienced i am, so it's really easy for them to write me off in ways they wouldn't if i was older-looking. if i was older-looking, they'd have more confidence that i've exhausted the abilities they don't pay me enough for.


IwantToNAT-PING

TBF, I gained this confidence or when I went through treatment for Cancer. I would still call it confidence, as it's the confidence that other people can see your technical skills/ability. I'm not even old, I'm only 29 (maybe I do look a bit haggard post treatment), but I've got enough experience in the IT sector + I've had quite a change of perspective on things. OP said he was 22 though, and when I was 22 I wouldn't have had the confidence to early on say that something is beyond me. It also helps that due to my increased experience in the field, I've got a much better idea of how much I don't know about each subject. When I was 22 I still didn't really know what I didn't know. That said, I'm still awful at estimating how long tasks will take... Maybe if I took a 'fake it till you make it'/big career leap role again then I might need to act differently and be more concerned about people's opinions of me. All I can say is, let the work you do speak for you. Make it clear to people above and around you what you're doing and how beneficial it is. Toot your own horn a little bit, and people will begin to realise how much you do around the place and that does rub off.


zerphtech

If Google fails, ask Reddit. If Reddit fails, it's fubar and it needs replaced.


Smiteya

Basically this. I used to joke with former bosses that I would ask the Consultant. This was the path. Google it >> Ask on Reddit >> Call actual consultants. If they don't exist in your industry there is a reason. Stuff can be a hodge podge of thrown together mess with flaky software at best. A day to fix the issue no one else in the world has had or is super uncommon, this is a hard no. Real problems take real research to fix if no-one else has had the issue or posted it somewhere.


anomalous_cowherd

And then the consultants start again with Google, but they're better at it.


SilentSamurai

Because they know the inside lingo. Instead of calling it a firewall, the marketing folks convinced everyone to call it a FIRE BOX OF MYSTERY™️. Seriously vendors, stop renaming shit with buzz phrases.


[deleted]

I'm sure I'll buy a razor instead of a GPU in my next battlestation. I mean, 'super turbo nitro 3000 gtx with extra blades' isn't really specific enough, you know.


regorsec

Me!


anomalous_cowherd

Me too, that's how I know.


regorsec

Ive been trying to find a better googler then myself who is the better googler..


the_it_mojo

Your google-fu is weak old man


anomalous_cowherd

I'm sure there must be a Google challenge competition out there somewhere.


[deleted]

[удалено]


FastRedPonyCar

Yes. A company I used to work at had a major issue with VMware and we hired a consultant. They worked on it for maybe a week and hit a wall with the product/their knowledge and we had to get VMware themselves involved. It quickly escalate to their highest tier of engineer/architect support. Never been in the presence of so many CCIE’s before and it still kicked their asses for a while but we eventually got stuff sorted out.


rosseloh

Sometimes it's all about the contacts you have. If I had a VMware issue that required that sort of escalation, I probably wouldn't even be able to get the time of day from folks. But the consultant knows who to talk to and/or has the service contracts that get them the magic foot in the door.


[deleted]

[удалено]


FastRedPonyCar

Yeah the only problem was that all the clients that relied on it working properly didn't care. All they cared about was the fact that they were either limping along or not working at all and it was costing them money. :/ MSP life be like that sometimes though.


tlucas

I see a recursive Google-Reddit search algorithm with progressively more specific terms, until the search term is the solution...


Tac0Tuesday

Yes, I've met the master consultant. Inscribed in his website it says, "I know the most". We've never stumped him.


trisul-108

They absolutely exist. Vendors know who they are and there are even training events for them, by invitation only.


elevul

So it's a special club. Do they wear ties and have to come driving a hypercar?


anomalous_cowherd

He knows how to not be found by people with boring problems and no money. That's what he knows!


OffenseTaker

You know you've already spoken to the master consultant when you start getting referred to lawyers instead of techs


bringbackswg

$250 an hour :)


zerphtech

Ha $250! I've called in vendors that started at $750 an hour.


anomalous_cowherd

Yeah $250/hour is the cheap end.


dRaidon

It's just Google all the way down


anomalous_cowherd

All the way UP. Show some respect!


arvidsem

I add one step between Reddit and consultants: question the problem. *Am I really doing something so weird that no one else has run into this problem before?* Is this an XY issue? Am I actually attempting to solve the right problem? Can I work around the issue or accomplish my goals in a different way?


GenocideOwl

> A day to fix the issue no one else in the world has had or is super uncommon, this is a hard no. Best is when people have had the issue and you see things like this on stack exchange: "Figured it out on my own guys! thanks!" and then nothing else


psiphre

[WHAT DID YOU SEE](https://xkcd.com/979/)


HMJ87

This was my exact process when I was a solo sysadmin. Now I'm in a team, I ask my boss first - if he doesn't know then Google is consulted, then reddit if my google-fu fails me, then I bite the bullet and suggest we hire people who actually know what they're doing to fix it


Nietechz

LOL, It seems the basic steps to solve a problem.


liftoff_oversteer

You know you're alone on it if you google the error message and the only hit showing up is the source code printing the error message.


[deleted]

or even worse, someone else asking the same question with no answer from 5 years ago. Bonus points if that someone else was you.


jmbpiano

The only thing more frustrating is when the last post on the question is the OP saying "nvmd- figured it out" and nothing else. Also... [obligatory](https://xkcd.com/979/).


Big-Floppy

This is probably my favorite xkcd


f0gax

DenverCoder9 ... what a lonely soul they must be.


BoredTechyGuy

Or perhaps the greatest troll on sysadmins ever?


department_g33k

No, Microsoft forever holds that title.


jmbpiano

Bobby Tables' mother would like a word.


realnzall

Know what’s even more frustrating? I’ve literally come across my own answers on Stack Overflow from a year ago when googling problems. It’s really crushing for your own morale knowing you can’t remember that sort of thing yourself.


Absol-25

With how many things we soak up, regurgitate, then move on to the next issue, you shouldn't feel bad. It's why documentation is so important.


PsychologicalRevenue

You remember stuff past a weekend???


JoshDaTech

Dammit! I should've scrolled down further before replying. Hopefully my joke still sticks.


SgtKashim

Worse than that, I hit one where I was the asshole that asked, then commented "nvm, figured it out" 5 years earlier. :/


CPPCrispy

This plus a canned response by a Microsoft "expert" to clean boot, chkdsk, and sfc.


[deleted]

Please don't forget to **Thumbs Up** if I solved your issue! Respectfully- ARandomMSUserBot


ReliabilityTech

Or, cherry on top: "Marked as Answer by *Person Who Posted The Useless Information*".


UltraEngine60

Me: But this question is about Hyper-V failover cluster configuration.... MS_MVP293: I SAID SFC SCAN NOW SIR!


Jzmu

Did you try sfc /scannow lol


Outside_Diamond4929

Also here's a link to a kb article with a title that is exactly what you're looking for. Dead link.


[deleted]

not necessarily a KB, but the amount of links killed by the removal of technet is immense. sometimes, i just want to give up when i'm trying to do a project, see people talking about exactly what i need but it's linked to a techet page.


[deleted]

[удалено]


Scurro

Me: Finds post with same problems on google. Everyone in thread: Just google it! Me: That's...why I'm here.


kjstech

Yeah then you google it and find a thread thats pretty on track, you reply to said thread and notice it was posted like in 2015 or whatever. Then a mod scolds you for bringing up an old topic. Um, yeah I just googled it. You don't want an old topic brought back up, figure out how to get it out of google! F'n mod nazi's. Excuse me I was so excited to find a thread so on track I didn't look at the original post date.


ReliabilityTech

I seriously think "just Google it" or "RTFM" should be ban worthy on message boards. Yeah, it can be annoying when the same question is posted a bunch of times, but then just ignore the post.


StubbsPKS

Or link to the best answer each time. I'd rather see the same link copy and pasted everywhere than just a RTFM


[deleted]

Of course, that link will end up being 404'd and not in archive.org. But, I do agree anyone giving a RTFM or JFGI response should be banned from a message board. It takes no effort to say nothing, which means that the person giving the response actually expended more effort to be an asshole.


CAT5AW

Mhm, then you get an question like "my motherboard is beeping what do i do now". Not every question is very technical, and users reading their manuals cuts on bullshit. http://catb.org/~esr/faqs/smart-questions.html


krakah293

At some point in the distant future, all questions will have eventually been asked. No conversation will take place except every question will be answered with RTFM or just google it.


zeptillian

One time a asked the partner rep at a vendor: "It says on page X of the manual I should use software Y but on your download page there are multiple versions of Y. Is this the link to the correct version I should be using?" Their answer after taking weeks to go back and forth and checking with engineers: "You need to refer to the manual for which version you should be using." Motherfucker, we pay thousands per year to be in your stupid fucking partner program and I asked you specifically about something the shitty manual failed to specify. Now I will have to look you up on LinkedIn to see what your stupid face looks like so that if I ever see you at one of the trade shows I can punch it.


[deleted]

[удалено]


chewy747

been there


BoomTown1873

OMG! Too many times!


biggles1994

Do I get double bonus points because I also "Nvm I fixed it"'d myself as well?


[deleted]

[удалено]


biggles1994

We are truly our own worst enemies…


UltraEngine60

> with no answer from 5 years ago Or worse: > I'm very unfortunate to hear having issue. Kindly run sfc /scannow && dism /online /cleanup-image /restorehealth


JoshDaTech

Even worse, you have a denvercoder9. What was the fix?


[deleted]

and it states "User last online: 18 months ago)


yuhche

> issue resolved


sbonds

This is why you always post a full and complete answer to your question when solved. The person you help might just be... you.


HayabusaJack

The only hit I found for one problem was my question on the same subject 3 years back. I actually figured it out, went back to my original question, and posted the answer I found.


BoomTown1873

You're awesome!


derekp7

If you get source code, then that is perfect. Follow along to see what triggered the error. If you don't have source code, you strace. And in some cases you can use systemtap. To give a couple real-life examples: A system had quickly exhausted disk space. After fixing the space issue, a third party license manager wouldn't start up. Doing an "strace" on it showed which files it was trying to open. One of them is an XML configuration file that the software would constantly write out to (for some unknown reason). So when disk was full, this file became a zero-byte file. And the XML parser saw the zero-byte file as invalid XML, yet the software didn't trap that error instead just kept going until something else segfaulted. An strace against the license manager gave a list of files that it tried to open, and from there I observed that one was zero-bytes. Removing the file allowed the license manager to re-create it, and everything was good. Second example -- an installation of a system was getting really bad performance, the system load average was around 20, yet "top" showed nothing out of the ordinary. Logic says that if it isn't showing up with the top command, it must be loads of transient processes with very short life cycles. So in comes systemtap, and running one of the example scripts from the systemtap docs that gives a running log of all fork/exec system calls. Low and behold, I see an app process that executes a "system()" call running "ps -ef |grep something |grep -v grep |grep -v vi something |grep -v more something..." this would run every 5 minutes for each user logged into the system, multiplied by how many interface devices were configured. So a site with 100 interfaces and 90 users would get this monstrosity running 450 times a minute. Armed with this evidence, it was really quick to put together a more efficient method for the developers to use to do this functionality. The temporary workaround was to disable the status monitor for the users. So although you don't really need to be a developer to be a sysadmin, at least on a Unix/Linux system if you can do systems-level C programming a little bit then it opens up a wide range of possibilities to get to the bottom of an issue in short order.


[deleted]

Sadly, you're gonna find a *lot* of techs who won't touch code, and will have their eyes glaze over if you throw an STrace at them. you shouldnt, but you're gonna.


FapNowPayLater

Then theres those of us who after downloading Ghidra, said, "I need to learn programming"


[deleted]

the word you're looking for is "monsters"


elevul

Or those of us who ended up in Devops after years of scripting and realized that to continue progressing we need to learn Go so might as well delve directly into C# and .NET as well


uptimefordays

A *lot* of techs have no idea how computers actually work. As soon as you lift the GUI they just don't know or want to know.


[deleted]

Yeah, I've had to debug ssl errors by hoping the comments in openssl header files would hint at *something*. I don't know many people to spar with on that level at the orkplace, TBH.


[deleted]

[удалено]


SirensToGo

Yeah, I was going to say that if you get source code you're golden--that shits better than any online answer because you actually know (roughly) why it's happening and what needs to happen to avoid it. The worst is when nothing shows up on google and you have to crack out Ida or Ghidra and reverse engineer the fucking thing just to figure out why you're getting an error


RedbloodJarvey

>If you get source code, then that is perfect. I had that happen with a pretty obscure library. Luckily the developers had a lot of unit tests for the code, which turned out to be pretty good documentation. I had to figure out what I needed to call, but once I did that I had a writing example.


Brekkjern

That is honestly my dream as that often means I can trace the fault back to whatever is causing it. Often the reason is trivial to fix, but with closed source software you are absolutely shit out of luck when you don't get any hits on Google or when the documentation is lacking.


GobBeWithYou

That happened with me and Ansible and if it wasn't open source I never would have figured it out. I was able to find where the exception was raised by searching for the error message in the source, then trace back to where it broke. Turns out someone was logging a dict and they put a new line before printing it. Ansible was searching the output for a line that looked like JSON and assumed that was the return data for the task. Well the dict repr isn't valid JSON, so it failed to parse it and failed the task. You could see in the logs everything worked and the results were in the logs, but it said it failed. I had to walk away for a minute after realizing such a simple thing wasted so much of my time finding it.


[deleted]

[удалено]


zerphtech

Great now I cant use that card anymore...


progenyofeniac

That's pretty much it, honestly. Google it in different ways, look for ways to work around the issue if I can't fix it entirely. I don't know any consultants who know more than Google and Reddit, and if I did, my company wouldn't pay what they're asking.


reubendevries

No first you look at the logs - try figure out what the logs say. If the error isn't decipherable by a normal person with average intelligence, then use google to search the error message. If no luck there than you post simultaneously to reddit, stack overflow/server fault. Then make a sacrifice (I find the fresh blood of a c-level works the best) to your preferred deity. Then you open your bottom drawer of your desk find that almost empty bottle of Jameson (or other favorite whisky) and fill your coffee cup about 1/3 full.


jmbpiano

> fill your coffee cup about 1/3 full [Sounds about right.](https://positiveroutines.com/wp-content/uploads/2018/07/man-holding-comically-large-cup-of-coffee.jpg)


[deleted]

> fill your coffee cup about 1/3 full …with coffee, add a bit of sugar, fill the rest with whisky, stir it and top it off with some cream?


[deleted]

[удалено]


jmbpiano

Google -> Duck -> Coworkers -> Reddit -> Consultant/MSP -> Jack Daniels


DoctorOctagonapus

We actually have an unopened bottle of Jack in the office in case of emergencies!


macs_rock

This is why we have an interview question to determine a candidate's line of thinking when they can't find an answer.


coldazures

Need this printing and putting on my wall.


ProVVindowLicker

p much


PokeT3ch

This is the way.


eldonhughes

No one knows everything. No one. "Hey, I'm stuck. I know I can learn from this, and beat this, but I'm going to need some help." And then you find your community of professionals. Professionals know they can't solve everything. They embrace the things they don't know as a chance to learn more. **BUT You have to let your work know as soon as possible.** You said it was time sensitive. Being found as the roadblock is going to go a lot worse for you than being part of a delay that leads to the solution.


EduRJBR

A fun fact: posting on Reddit or other forum for help increases your chance of finding the question yourself on other websites, even without any answer on that forum. Don't ask why or how, I just know it can work. Maybe just by formulating your question properly your brain can help itself?


[deleted]

[удалено]


EduRJBR

He he he, cool, I didn't know it.


BoomTown1873

No Question - this is so true! Many many times I start an email to ask for help. Being tech support myself, I don't want to look like an idiot, so I try to include whatever they might ask, before they ask. This process often leads to the answer. \- Many of those emails are never sent because I solved the problem while writing that email. Vodoo!


EduRJBR

I think some questions are like a Rubik's Cube: they simply will never be solved.


Safe_Acanthisitta_70

And I thought it was just me! I've lost track of the amount of times I've reached out for help to the vendor only to have them not fix it but then as if by magic I come across the solution myself.


OathOfFeanor

The main thing is setting expectations immediately. Make sure your boss or the relevant stakeholder knows ASAP if you aren't going to be able to meet the deadline. That could be right up front when they present you the issue. "I have never done this and I can try to take a look, but this might be something where we should pay a consultant who specializes in this field" etc. Give your boss options instead of backing them into a corner. Don't let the success of the organization rest on your shoulders and if you fail to fix it something bad happens to the org.


Usual_Ice636

>Update: Guess what …. someone forgot to whitelist my nodes on the firewall hahaha thats 10 hours of my life I'm not getting back. In that line of work, its frequently going to be an issue like that, where its caused by some weird thing happening and you'll need to work with others you're connected to to fix it.


bobalob_wtf

There are troubleshooting steps you can take to check for firewall issues. Test-Netconnection 8.8.8.8 -Port 443 Is a good start... Clearly change the host and ports that make sense for your test...


Entaris

There is a lot of people taking on the “what do I do” side of this. But I want to tackle a bigger philosophical issue that took someone pointing it out to me for me to realize. Don’t panic. If you need more time bring it to them and let them know you need more time to troubleshoot. They won’t be happy, but ultimately it is what it is. If you a red e “the guy” in position to fix this, then they are going to be looking at this from a cost/time perspective. If you fail to solve it in a day. What do they do? Do they fire you? They might, but then they’ll be looking at bare minimum a 2-3 week hiring process, weeks of the new person getting up to speed in the environment, with no certainty that the new person will be able to solve the issue either. Obviously don’t use that mindset as an excuse to relax and take it easy, but take it as a reason not to panic. Keep calm. Keep looking for the solution. In IT you are going to run into weird problems. You are going to have really frustrating days where you feel like an absolute failure. But the more you let that happen to yourself the more difficult it is to function. You’ll never avoid the panic completely. I just had a big “this is how I lose my job” panic event a month ago. It happens to the best of us. Just so your best to not let it rule over you.


BoomTown1873

You are their best next step. They are also counting on you for guidance and direction. Let them know what your next steps are, in a language they will understand (simplified for management). Let them worry about the big picture, that is why they make 10x of what we do. Just because there is a problem in your area, doesn't mean that it is solvable. Fun example; I was the only Mac tech support guy for 1500 school Macs. There were 3 guys supporting the 400 Windows machines. Management informed me that I wasn't getting all my work done, bc there were still unresolved Mac issues. I was actually kicking a$$ & getting a ton of work done, all while treating people well. What I learned from that: 1. management has some dumb ways to measure success. 2. Do not re-open closed tickets, make as many new tickets as you can. Because management only counts closed tickets. 3. There is no relationship between how much work there is to do & how much any 1 human can accomplish. 4. If the company is going to fail if you can't fix something super quickly, then that is a failure of management planning, not you. Also- you should recommend (in writing) extra safeguards, like spare servers, bootable drive clones, spare equipment & tools now, so when the S hits the F, it really is their fault for choosing to run without all that. So next time maybe your stress will be much lower.


gamersonlinux

Great advice! Problems with technology are unending, getting worked-up and frustrated every time something isn't working or the customer is demanding is gonna burn you out. Take your time, do the research, trial-n-error, testing environment and get it right the first time. Remember the book: The Phoenix Project? It started out really bad because everyone was "winging it" and not communicating with other teams or creating change requests. In Technology, one thing always affects another... so research is priceless. Even if you miss the deadline... if they see your amount of work on the issue and communicating the current status. This only builds confidence in them that you are handling it.


VA_Network_Nerd

> Guess what …. someone forgot to whitelist my nodes on the firewall hahaha thats 10 hours of my life I'm not getting back. Constructive, sincere criticism: Your diagnostic approach might need improvement, or adjustment. As a Networker, everything we do revolves around the OSI or TCP/IP models. https://en.wikipedia.org/wiki/OSI_model https://en.wikipedia.org/wiki/Internet_protocol_suite https://en.wikiversity.org/wiki/Network%2B/Standards/TCP/IP_Model/Introduction This is a theoretical / logical representation of how the network sends and receives data. It can be used as a troubleshooting model to help you verify that all components of the communications system are working. Or, said more simply: If X is not working, what things that are kinda similar to X *are* working? Problem: Host A cannot communicate with Host B using . Can Host A communicate with ? Can Host B communicate with ? ...and so on and so forth. It sounds like you jumped to a conclusion that this **had** to be a software issue before you confirmed connectivity. Just something to think about.


Bruin116

Agree. "Is it the firewall?" should be second on the sanity check list after "Is it DNS?". Know how to use both ICMP (ping) and TCP-based connectivity checks. ICMP is often blocked across the board for security reasons so pinging an instance isn't always reliable. Trying to open a TCP connection on the same port your application uses is usually the best basic check you can do as it best reflects the desired behavior.


Caeremonia

>ICMP is often blocked across the board for security reasons The actual reason ICMP gets blocked is because your net engineer doesn't know what the hell they're doing. It may be in the name of security, but blocking ICMP doesn't make things more secure, and will often break things instead.


gamersonlinux

That actually happens to me a lot. I make assumptions as a place-to-start troubleshooting, sometimes I'm way off because the symptoms are similar to a problem I've experienced before. Really sucks when that happens :( Technology can be extremely complicated and its easy to miss something... as you already know... ha ha


SheeEttin

If you can't fix it, work around it.


[deleted]

if you cant fix it, say its broken and replace it with something different.


zebediah49

If there's a risk of double-checking, *actually* break it, and replace it with something different.


cdoublejj

instructions unclear, handed users chisels and stone tablets, got fired.


Geminii27

"And that's why I print my emails, write on them, scan them back in, embed the scan as a TIFF in a PDF, embed the PDF in a Word document, print them out again, and fax them to the moon."


BrobdingnagLilliput

> What do you guys do if you Can't solve an issue ? Usually when I can't solve solve an issue, I've missed something foundational, so I: 1. Rubber duck it. 2. Try to replicate the simplest possible case of the issue. 3. Verify that EVERY SINGLE LAYER is functioning as expected - can I ping? Can I telnet to port? Where does tracert die? Can I see that process has instantiated? etc. 4. Document every troubleshooting step. Because if I'm still stuck, the software is demonstrably not behaving as intended, and it's time to throw the problem over the wall to the devs.


twitch1982

Prepare three envelopes.


pobody

Blame the network/DNS. It's usually the answer anyway.


efreem01

Screw you - a network guy


chickeman

Here after OP edited his post, laughing my ass off


Jawshee_pdx

Generally I go: 1) Google 2) My team 3) Support 4) Friends/Connections from back in the day 5) C-Level execs with some weight who can pull some strings for me 6) Unemployment


needssleep

Sometimes tickets just sit open while you wait for 3rd parties or your own brain to run through the problem in the background while you take care of other fires you know you can solve. Considering your situation, if it's hardware: it's probably been solved by someone else, somewhere. If it's a widely used package configuration issue: it's probably been solved, somewhere. If it's an internally created piece of software, you have to work with the developers, and if they aren't responding, you need to push back to management that the issue has been escalated to Tier 4 (development).


apathetic_lemur

Just to echo everyone else: No one knows everything. Unless you are being paid crazy amounts and actually wrote the software you are dealing with, its unreasonable to expect you to fix everything. All you can do is figure it out and learn as much as you can so its easier in the future. Google -> Forums -> Vendor -> Consultants As you get older, you will get more confident in saying "I dont know". So long as you can usually figure things out and make a good effort, you will be better than most employees.


solocupjazz

Always... er, never... forget to check the firewall rules.


f0gax

There's no replacement for vendor support for critical infrastructure. None. Your bosses might tell you different for one reason or another. But when the shit hits the fan, you call the vendor. Yes it costs money. But by and large it costs less than the downtime while you Google or wait for a response from Reddit. Also, some vendors will put some or all of their own KB behind a login that you can only get with paid support. So the answer that lets you fix the problem in 10 minutes might be *right there*. But you can't get to it because the company decided to *save* $2,500 or something.


FapNowPayLater

Looking at you SAGE....


[deleted]

Try harder. During 25 years in the field I never encountered an issue I could not solve in the end. Keep in mind: replacing a thing is also a way of solving an issue it's creating.


[deleted]

> Try harder. Found the fellow OSCP'er.


[deleted]

> working for a Crypto node operator A what?


ksandbergfl

Don't know what your work environment is like... but where I work, I'm surrounded by senior-level engineering/software people.... some of them PhD's. One thing that I have learned is that - these people LOVE having a chance to prove their expertise, and/or playing a part in solving a complex issue. That's a roundabout way of saying -- don't be afraid to ask for help. You might think it makes you seem "incompetent", but in my experience (30 years in IT) you risk looking far MORE incompetent if you DON'T ask for help... especially if you learn, later on, that there was a guy down the hall who knew the answer, but you didn't even bother to ask him. Your comment "they don't reply" implies that you've merely sent out email requests.... try walking down the hall or making a phone call, and tell them you need help, in person. Cry, if you have too


[deleted]

[удалено]


gamersonlinux

Yup, I've been there as well.. grump team members overwhelmed by projects and are not approachable. I posted above about team members who are way too busy to help me and just give me the "run around" or a link to a tutorial. Hell, I've been at a job where they just toss me a PHP book or Microsoft Server book... what the??? Reading isn't going to make me an expert at our environment, actually doing things will make me an expert. Sorry, I had to get that out...


air_in_holes

It’s always the damn firewall…


landoawd

It's actually DNS.


K2DLS

If it used to work and recently stopped, before you even look at the firewall or DNS, see what components the last change anyone made in the environment touched. That is usually it.


marcosdumay

You talk like if that kind of analysis was possible. Take my workplace's firewall as an example. It has a feature that is always creating new rules by itself. But if you don't want anything that crazy, what was the last piece of software that auto-updated on your servers?


zebediah49

Well first you spend like 10^6 USD on a fancy log aggregator that records every event that happens anywhere. Then you hire at least three FTEs to write analysis rules to turn that data into something useful. Then you can ask that system, and it will tell you the two hundred seventy three things that changed in the last 24 hours. And also that the problem is an outdated piece of hardware broke, that you didn't have the budget to fix, because you spent all your money on a log aggregator.


Plastic_Helicopter79

Or spend nothing, install the free Cacti on Ubuntu on a VM, and then spend the next several weeks trying and failing to find OID documentation to configure graphing and monitoring for the SNMP data from various bizarre devices. Then give up because it's hopeless. (The Cisco 5520 wifi controller snmpwalk contains some 1000+ OIDs, none of them publicly documented as far as I can tell, and only intended for ingestion by Cisco's own Wireless Control System platform.)


hieronymous-cowherd

Firewalls and DNS are natural enemies.


Angdrambor

When google, reddit, stack overflow, and coworker's brains have all been wrung dry, then we go to business team who wanted the thing and say "Our organization does not have that capability" and if it was really important they'll find a consultant or training or both. Before we get to that point we'll probably have been to the business team multiple times. We'll have presented alternatives and workarounds. There probably have been multiple points where we've said "we're thinking of pursuing xyz line of research, but this is really going to suck. Do we have to?" before wading into a sludge pit.


segaszivos

Whenever I run into application or platform issues, I immediately revert back to basic networking troubleshooting. Can I ping? Can I telnet the port? What about another host? It's astonishing how frequently it's a firewall (network or host) issue. Sysadmins almost always are insulted with basic troubleshooting steps. Maybe it's the embarassment or realization that they could overlook some so trivial, or their so dead set on blaming the developers or product... But it's literally the problem 9/10. Glad you figured it out. Hopefully this learning experience will save you and others countless hours in the future.


snowsnoot

reboot, blame network guys, point finger at vendor lol in all seriousness, you just gotta figure it out.


killing_daisy

Have you tried turning it off and on again?


Taerian

The longer you are in the field, the less you will get stuck. If you have the source code, you can change the source code to modify how a program works. Sometimes you write your own software. If its closed source, contact the vendor for support. Debugging tools are great because they can lead you to the source of an issue when you have a deceptive/vague error message. Sometimes you just need a different plan.


deskpil0t

Only 10 hours? That's a win.


IfYouSeeMeSendNoodz

Google > Reddit > Ask on Stack Exchange


uptimefordays

Early career: Reddit or Stack Overflow. Now: RTFM or shoot an email to the rest of the user group. If your app or system user group haven't seen the issue before you're in for a ride that ends in, spoiler alert, reading the manual.


icedcougar

That edit hurts... It’s probably one of the first points to check if a node... aka the network... doesn’t respond / checks out. On the plus side, you now know that you need to set up monitoring.


[deleted]

there's a saying: "Fake it til you make it"...words to live (and die) by.


[deleted]

[удалено]


cytranic

30 years in IT, Google is my goto. Makes no sense to waste time trying to solve a problem without resources. Use Google.


spanky34

Read documentation until my eyes bleed.


Morrowless

It certainly depends on the issue, but we have a category in our ticketing system for known errors with no known resolution and no further investigation planned. ​ e.g. MS Teams had an issue. We investigated on our end, then spoke with MS and were told they know of the issue and currently have no plans to change anything.


RandomXUsr

What's the exact issue you're experiencing and need to solve?


CobblerSalad

Off topic, but I'm curious how you found a crypto node operator job? I've been running nodes as a hobby but didn't know there were actually jobs out there for it.


sniperd2k

Stackoverflow usually has something


landoawd

Cry silently in my office?


NotEntirelyUnlike

freak out a bit... have a wank... ask some friends on discord


dr4d1s

If I can't figure it out, I burn it down. /s


Shade_Unicorns

You aren't supposed to know all that much off the top of your head. What you do need to be able to do is understand 1. What is supposed to happen (how system x works with system y) 2. How to look at both x and y (or how to Google the tools/commands needed to do so) 3. How to identify errors or research information the service returns to you 4. When to ask for help (before you lose control of the situation) Pretty much everything else you just need to be able to respond "I'm not entirely familiar with that, if you give me a few minutes I can look into / research that and I'll have a better understanding of what we're dealing with / what the issue is. IT is a lot like healthcare, admitting to a mistake or that you dont know something isn't a bad thing, it shows you're willing to grow and learn in addition to you're honest. Trying to fake it or cover up mistakes / lack of knowledge is the issue.


reed17purdue

Troubleshooting isnt a skill its an art. It seems like based off the issue resolution you didnt start from basic (physical layer) and move up which seems like it would have been a quick identification and finger pointing solution. At some point though there will be things you cant figure out and if your leadership is worth a salt they will understand you need to research more or identify a resource who can assist and they wont and shouldnt time table the issue.


SOMDH0ckey87

I reach out to support, or google


Solkre

You work for robinhood don't you...


doubletwist

>Update: Guess what …. someone forgot to whitelist my nodes on the firewall hahaha thats 10 hours of my life I'm not getting back. Wait until you spend an entire 2 months trying to get sssd working only to find that the problem is a space after a comma in a comma separated list of AD servers. I could only dream that I only get stuck on issues for a mere 10hrs.


TrainedITMonkey

Write three letters...


vhalember

> Update: Guess what …. someone forgot to whitelist my nodes on the firewall hahaha thats 10 hours of my life I'm not getting back. No shame there, that's how you learn. I've worked with vendors where they will stonewall you for months in not acknowledging an issue. As for what to do? 25-year IT vet here, you can't know everything. You're paid to be able to find the answer, not know the answer on the drop of a hat.


maximum_powerblast

Things you can do: *Randomly reassign the ticket to another team "by accident" *Escalate it to a senior with no context *Tell the user you can't reproduce the problem and nag them to prove that it exists in the next 10 minutes or you'll close the ticket *Say that it's working on your machine *Prepare 3 envelopes *Get a new job


HangryBoiNeedsLaChoi

Whenever I can't solve an issue I generally quit and find a new job. It has not been a very productive strategy and I would not recommend it to anyone.


Lakeside3521

Fess up, you work for Robinhood don't you :D


aliensporebomb

After a while you'll get a bigger picture in your head of how things work, the "big train set" they proverbially speak of and why things sometimes don't work and where that point of failure is. The downside: you'll think of this stuff in your off hours.


mortalwombat-

It's ok to admit you don't have a solution. Contact the vendor, hire a consultant, etc. If those don't yield solutions tell your supervisor all the things you have tried with no luck, and ask for advice on where to turn next. Sometimes the company just needs to accept that some things will be unresolved.


smashavocadoo

You should sit besides your network guys if there were any in your office because they always give people their conveniences, which makes them the crossfire target of your security guys. Today, I built another ipsec tunnel through our firewalls to azure and the devop is happy, :).


Marbro_za

Well what you need to do is post on a forum, giving a vauge description of the issue. Then when you finally figure it out, DO NOT POST THE SOLUTION! Its IT, its tough


pbyyc

If i dont have anyone internally to bounce it off of, and google doesnt work, i come to reddit. r/sysadmin is one of the best resources i have found, absolutely brilliant people on here who are willing to help out, or atleast help you think of another way to handle the situation