National Usenet Breakage Day

[serinde]

...Or something. I didn't seem to get the memo, so I'm guessing.

Prequel: the primary reader crashed Friday night. Rebooted, no big deal. However, since then the natives have been woeful, noting that posting times are increased--that is, you post the article and it takes anywhere up to a minute and change for your newsreader to come back and get on with its life. At first I thought it might be related to said crash, but it was happening on both readers, and with any news client, including just talking directly to the NNTP port. (Which I still remember how to do, because I am awesome.) And then sweh reminded me that, even though it doesn't look like what's happening, when our users post to the readers it doesn't actually post the article per se; it gets passed through shogun and then fed back to news-xfer which actually posts the article, so logically the nose goblins must be on news-xfer. But how does the article get there since it's not through the usual method?

I keep expecting someone whose job title officially says "sysadmin" to deal with the deep magic of news-related troubles, but that keeps not happening. I am in fact the one with the biggest INN clue except for El Jefe himself, and although he had a wizard's knowledge of it in the day, he hasn't actually touched it in years. So here am I with the leader badge.

The hunt goes through dark places, because our news setup is weird and mystical and somewhat ill-documented: that is, parts of it are meticulously documented, and others were Not Got Around To before the people who knew them left. Happily for me, many of those people are still around my world. So I got jdev's help on ktrace and ktruss, and 5tonsflax kindly spent 40+ minutes on the phone with me as we unravelled the silliness together. And as often happens, the solution is very simple once you know where to look: the history database had gotten frightfully bloated (even for modern history databases), because expire hadn't run in $SOMETIME, because for some reason /news/db had become owned by root. None of the files in it, mind you, which I woulda noticed. Just the directory. Which is the more odd because the mount point in the dist structure is correctly permissioned, so this breakage should have been fixed at any of the times we've disted news-xfer in the last $SOMETIME (including this morning).

So, fixed that, was about to run expire when I realized there is probably not enough space on the partition. Phoned El Jefe. He thought there was, because he thought that the re-written database would be smaller. I didn't think it would be smaller enough. What a stupid time to be right... It's now running and dumping the new copy in a really silly part of the filesystem that nevertheless has scads of space, and hopefully it will finish in time for me to make sure it is all okay before I GO HOME.

Also, some Usenet performance artists are splashing their poo on various walls, and making a clumsy attempt to joe-job Supernews with it. Anyone know if "usenetserver.com" takes action against this kind of crap? And had to explain some basic newsgroup functionality relating to this to a guy who says he's been moderating for 11+ years. THEN YOU SHOULD KNOW WHAT I AM TELLING YOU, BUNKY, AND YOU SHOULD NOT ACT LIKE YOU KNOW MORE THAN ME BECAUSE CLEARLY YOU DO NOT.

Comments

[arkham1010.livejournal.com]

[syringavulgaris.livejournal.com]

Yes please.

[syringavulgaris.livejournal.com]

OH I SEE WHAT YOU DID THERE

[arkham1010.livejournal.com]

quick on the uptake today, ain't you. ;)

[ronebofh.livejournal.com]

Or, try this icon.

[reddragdiva]

"I keep expecting someone whose job title officially says "sysadmin" to deal with the deep magic of news-related troubles, but that keeps not happening. I am in fact the one with the biggest INN clue except for El Jefe himself, and although he had a wizard's knowledge of it in the day, he hasn't actually touched it in years. So here am I with the leader badge."

Welcome to sysadminhood! Here's your accordion.

[tskirvin.livejournal.com]

I'm sure I've got a self-written HOWTO around here somewhere...

[reddragdiva]

This is what internal wiki pages are for, i.e. notes to self that are theoretically for everyone.

[syringavulgaris.livejournal.com]

I'm supposed to be building that. HA HA HA

[reddragdiva]

1. Install MediaWiki. (Use Leengux box. PITA on anything else.)
2. Write it.

You're supposed to give the wiki to the n00b to write - then they know what needs to be in it by their stupid questions.

[tskirvin.livejournal.com]

Actually, it wasn't too horrible on Solaris.

./configure --with-perl --with-openssl=/usr/local/ssl --enable-largefiles \
        --prefix=/usr/local/encap/inn-2.4.1
gmake
sudo gmake install
sudo gmake cert
   [certificate was specified to news.ks, this will move as the server moves] 
cd /usr/local/encap/inn-2.4.1/
sudo chown -R news:news .
sudo mkdir base; sudo mv db etc log run spool tmp base
sudo ln -s /var/spool/news/db .
sudo ln -s /var/spool/news/etc .
sudo ln -s /var/spool/news/log .
sudo ln -s /var/spool/news/run .
sudo ln -s /var/spool/news/spool .
sudo ln -s /var/spool/news/tmp .

The config-etc stuff is somewhere else...

[reddragdiva]

I meant MediaWiki :-)

[xlerb.livejournal.com]

Hey, there is some actual documentation. For example, there's the document I wrote on a certain detail of the news system, insufficient appreciation of which will cause user whining exactly when such is least needed, having extracted the information from the machine relics by blood sacrifice, as per usual. And in its hour of FSVO-need (a tangential part of the matter in the original post, though not mentioned there), it failed, because I'd foolishly assumed that a certain piece of infrastructure would actually work, instead of (apparently) not.

I'm not entirely certain that a wiki could have helped there.

[xlerb.livejournal.com]

And that might help for INN in general; but scrying the mysteries of the local configuration is a skill that must be learned (though some have more natural talent for it than others), and there can be no substitute.

[tskirvin.livejournal.com]

This is true.

It is at least handy to figure out which files I need to edit on a new INN install, and in which order...

[syringavulgaris.livejournal.com]

Fuck the accordion, where's my sysadmin salary?

[reddragdiva]

For that you'd use your considerable experience on your resume and, ahh, not work for an ISP. They don't pay so well.

[msg1138.livejournal.com]

OMG I HEART ICON KITTY!!!!!1!

Actual post hurts my brane tho.

[lillilah]

Ugh. That does sound typical though. It seems for almost every problem there you can say, "We've got this insane setup, and some random part that no one has looked at in a while got screwed up." Oh well.

[arkham1010.livejournal.com]

[syringavulgaris.livejournal.com]

What "fail"? I succeeded. *sneer*

[arkham1010.livejournal.com]

i dunno, i just like the picture. :)

[xlerb.livejournal.com]

because for some reason /news/db had become owned by root. None of the files in it, mind you, which I woulda noticed. Just the directory. Which is the more odd because the mount point in the dist structure is correctly permissioned, so this breakage should have been fixed at any of the times we've disted news-xfer in the last $SOMETIME (including this morning).

UI: There are files in /dist/etc that say, among other things, which directories do and don't get sent over, for each class of host (and, if so, whether or not to nuke files that aren't there in dist; this is called “static” (do nuke) vs. “dynamic” (don't nuke)). It seems not implausible that the entirety of /news/db might have been excluded entirely.

Also, some Usenet performance artists are splashing their poo on various walls, and making a clumsy attempt to joe-job Supernews with it.

Oh, I saw that. They threatened to make a froup I read “unusable” with spam floods or some such, but I didn't see anything else out of them after that. (They also cited their alleged earlier destruction of sci.crypt, which I haven't read in years, but given its density of cranks there are certain comments I could make along the lines of its not needing much help. (Which comments would not be entirely fair, but.))

[syringavulgaris.livejournal.com]

It seems not implausible that the entirety of /news/db might have been excluded entirely.

Indeed, /news/db is not listed at all. So distlocate is a lying whore. Thanks for ye pointre.

They threatened to make a froup I read “unusable” with spam floods or some such, but I didn't see anything else out of them after that.

I noticed a few cancels going by related to this fun in the course of yesterday's spelunking, but it looks like sci.crypt itself is pretty well inundated.

[xlerb.livejournal.com]

There is a sense in which distlocate isn't entirely a lying whore: the initial dist of a host (psync -i otherbox:/mnt, which is not a thing one uses very often, especially if one isn't Brian or me) will copy everything.