Program/website to view really really, and I mean big log files?

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0
Hi I have a 200GB log file I just finally downloaded - took my a good few days to do it and was wondering if there's a similar program or website to this: http://www.readfileonline.com/ which not only opens any sized text based file but also gets them from FTP servers and the like as obviously I am not keen on downloading another 200GB log file just for viewing purposes..... -.-

Also I would like to view the log as whole rather than it split into chunks of 10MB(which is what that website viewer seem to max out on) and go through 20k pages; I prefer to just view the whole thing without clicking next button.
 

Ralston18

Dignified
Moderator
Not really sure about the details of your situation per se. But your requirement is just to have one continuous "scroll" as I understand your post.

My first thought is that some demarcation ("chunks of 10 MB") is necessary simply because your computer, storage, software, website, etc. may not be able to handle "single viewing of such a large file.

If the data is a log file you might be able to download the log file into a database with each log entry being one text record in the database. Or some field structurer like Time, Date, Entry....

That said can you provide more details about the data itself?

Then you would have the added ability to search and sort - depending on the log file format.
 

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0
Sorry for late reply, been busy....

Yeah something like that an if it could read straight of the FTP server, that would be nice as well as that would save me from downloading the file onto my computer, which requires me waiting for it to be downloaded(luckily I'm on unlimited data so downloading 200GB is no problem(though on slow ass ADSL2+ 1MB down it is for me as I don't like waiting and leaving my computer on overnight for a few days for this to happen :/), but that might change in future when I move houses, etc.) and *then* read it from there.

ok fair enough. But what about in chunks of 100MB, or even 1GB? Surely it can read these length logs, can't it? I know locally, if yo u want to view a relatively large log/txt file, you need to open it in wordpad.........as notepad apparently can't open these large files....

Which would require me to download the file and spend the time to sort this out...?

Just a log of a game server I have up and running, or did anyways....apparently the problem my Game Service Provider (GSP) has said is that they will kick me off if I ever use up more than hundreds of GB for a log file.....mind you, I only had two main log files that was still there - the 200GB was which I was gonna delete after confirming I didn't download a corrupt file and the actual log file that was created whilst the server is still running - I'm guessing it must have got pretty big because they've told me I have 700+GB worth of what appears to be log files and that it was damaging their hardware blah bla bla....

I went to go ahead and download the whole 200GB file(this was maybe a few or so days before this happened), took a while so I could scope out what could be bloating the file to the point of crashing the GSP or at least causing problems for them for them to threaten to kick me off their list if it happens again. Apparently it's happened several times but I didn't know that.....and I'm guessing this could be the causes of these crashes, but GSP has it setup so that there's no downtime and resets server if it ever crashes, the problem with this is that it over writes the same log file so I am unable to retrieve and view it before deleting it to see what the problem is....

Anyways, so I go to login to check for myself if it's true or not and to maybe get that 700GB file or read somehow.....hahaha.....but they already locked me out so I couldn't do anything...so after a while, we agree on me fixing it up and that they'll unsuspend my account/service so that I can do it. ...and of course they reinstalled the service so lost some of the mods and stuff, they could have warned me....but ehh.....but at least I have a backup of these on my computer to re-upload so it wasn't that much of a loss....

So yeah, here I am trying to find a solution...and thought maybe the solution was to read the huge log file....for any errors....unless you guys have a better and quicker idea, in which case I'm all ears....?

Well I can already search, I only need to read, not write anything further to it. So Notepad or a log/txt viewer is enough to search for anything and let me view the lines for any errors.... Just that noneof the readers I have can open it, and the only reader tat opens it, can only read in a maximum of 10MB chunks and since this is 200GB file, I would have to flip through 200k pages.....so me thinking a one time scroll through the entire thing with crtl+f searching for anything of relevance and skipping the bloat lines......could work......might take a full day to go through that I guess......but if it means finding any errors and causes to these silent crashes that I mostly don't notice because I'm not always monitoring my server 24/7 as it runs.....then I was gonna give it a shot.



I guess read it...? hahaha :p I don't know, got a better idea of tracking down errors than go through a 200GB log file specifically looking for error like lines...?

Actually the initial idea of reading the log was to see how popular my server was in the public and that's it. Now that my GSP is pretty much forcing and bugging me to fix this or else they'll shut it down for good, I got back to this again and yeah....

Anyways, that's what logs are for right, reading, ye?
 

Tumeden

Commendable
Oct 15, 2016
31
0
1,590
4
What game are you running a server for? Sometimes people develop small applications like this for more popular games such as Minecraft for example. I've managed dozens of minecraft servers in the past and had an application to read very large log files which split up different errors into different tabs to easily find them. Might be able to help find something for you if i knew what type of game server you are running.
 

randomizer

Distinguished
Moderator
Jul 24, 2006
882
0
19,260
62


Logs are for many things but nobody does logging just for a bit of light reading. The only time I read a log is when I'm watching new entries coming in to see a problem logged as it happens, and only when I'm expecting it to happen. Unless a log is exclusively an error log it is almost entirely noise that I don't care about. I have both file and database logs and I almost always search them rather than read them.

Of course if you're going to search the log you need to have some idea of what to look for. If the log is a well known format then you may be in luck because there are probably applications that understand it already. If it's some custom mess then it might be worth splitting out a few GB from the end of the file to do simple keyword searches to see what an error might look like. Don't use Notepad as it is hopeless at reading files beyond a few tens of MB and even worse at searching them. Notepad++ is a better option for searching small to moderate files. Once you've got some idea of what to look for you can search the whole file. You'll either need need to split it up or use something that can scan the file (line-by-line or in chunks) without loading it all into memory. I'm not aware of any options for the latter on Windows but there probably are some. My go to option would be grep on Linux which you can use on Windows via Cygwin (but it will likely be much slower and it's another thing to get your head around).

As a side note, I'd consider enabling log file rotation if it's supported by the server. At the very least try to turn down log verbosity so that you don't capture as much noise.
 

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0
Killing Floor 1

Oh yes, if I can find something that can automatically sort and split this out as the log is been written to, in different files, that'd make my reading and searching that much more easier and quicker! Or at least have a program that grabs the log file and then sort it out between bloat, player joins or and map starts (so I can check the popularity) and actual errors I might be interested in.

Ok yeah fair enough. This log, I would expect pretty much bloat lines of stuff that's just taking up space, like pingbacks and stuff...



Well here's a sample log from my now up and running dedicated KF server(that's not from the 200GB log file I wanted to go through, heh. - this one's fresh, or at was fresh a couple or so hours ago): http://pastebin.com/Gc2fZeJf

That's just from a few games of KF when I had it open to see if any errors would pop out. From that, let me see......

Code:
Warning: Failed to load 'Class ScrnDoom3KF.Patriarch': Failed to find object 'Class ScrnDoom3KF.Patriarch'

ScrnDoom3KF: Unable to load end game boss class: ScrnDoom3KF.Patriarch
would be an error of interest.... I guess I'm looking for "Warning:" and "Error:" prefixed lines And I suppose anything else that kinda looks like it might screw up(or is screwing up) the server because something's missing or corrupt or whatever.... I don't know really...hahaha just you know anything that looks like it might be an error/warning I need to address or else bad things will happen...hahaha I suppose if any of yous run a KF server know what I should be looking for, then I'd be happy to be given some guidance on what exactly to look out for. Most of it looks like bloat lines that I shouldn't be worried about...?



What's log file rotation, is that something like backing up the log file and then have the server, if it crashes, write to another, but similar log file? So for example, UCC.log is the start, then if it crashes, it writes to UCC1.log and then UCC2.log? or something else, explain? Or do you mean that it writes to multiple files for logs? Or writes to this log file for a week, then the next week it writes to another log file? So UCC1.log, UCC2.log, etc.?

Well I would turn down all that junk lines but not sure how.....

OF course there's a 65k character limit.....to post on this forum.....
 

randomizer

Distinguished
Moderator
Jul 24, 2006
882
0
19,260
62
In the interest of preserving people's mouse wheels I've put the logs up on Pastebin and edited your post with a link.

You are right that most of it is noise. This is quite a verbose log. I would look for "Error:", "Critical:", "Fatal:" and anything else that sounds bad. I wouldn't be too concerned with "Warning:". If the server follows common conventions warnings are usually not serious but may still help diagnose some minor issues. For example, you may log a failed user authentication as a warning because it doesn't prevent the application from running but may be useful if the user asks why they can't connect. Look at warnings after you look at real problems, because there will be far more warnings than errors, and it's unlikely that you will be able to do anything about most of them. Do you actually have a problem that you're trying to fix or are you just looking for a problem that might not even be there? The only problem I can see right now is that you're logging too much.

Log rotation usually involves writing to a file until it is a certain size or a certain age (eg. you may have a file for each day or week), then archiving it and starting a new file. After you reach a certain number of archived files or if the archives reach a certain age they are deleted, starting with the oldest. If there is a problem it probably won't take a month to notice, so keeping old logs around forever is usually pointless. I have no idea if the KF server supports rotation.
 

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0
Oh, looks like they've changed the layout of TomsHardware quite a bit....looks like it's more suited to Windows 10 now....lol Not that I'm complaining, but I still preferred the shiny look(windows XP to 7) as to the flat look in windows 8, 8.1 and 10 have....

Hahaha, and that wasn't even most of it....maybe an 1/8th(or maybe not even that, probably smaller 1/32 or something, the scroll bar thing was pretty small like a square instead of the usual rectangle and viewed at 1920x1080 - minus the taskbar space that's take up at the bottom, so 1920x900 or something; speaking which that's not a lot vertical viewing space.....I should probably get cheap 1920x1080 and have it in portrait for 1920 pixels of vertical viewing area which also means less scrolling is required!) of it(I had shutdown the server, downloaded the fresh log file to see what's going on and then restarted), though with that kind of length, that's probably nothing compared to the lines in the 200GB one.....haha

mmmhm...


So you suggest I wait until I actually have a problem before going through the log and then find any problems? So essentially, don't bother reading that 200GB log file I have until I have problem(or at the time)?

Initially, I wasn't looking for problems, only wanted to read the log to see how populated the server gets. It would appear there is a problem, trying to find out the cause of the overly large log files I've been recently getting. Apparently it's abnormal to have more than a couple of GBs of log files in *total*......the guy said that last log file that apparently filled 700GB of space(we are only given 1TB SSDs - I was expecting maybe a RAID setup of HDDs and SSDs together therefore assuming it can handle large servers and log files, but nope; just a single SSD) so that is quite a bit of space to take up) only took so within a week! That would mean it would fill 100GB a day! So I suspected it may be the leveling perk maps I have had so I removed them completely from the server as I read some from the log chats that it apparently crashes the server on one of them if you do too much of [this, I forgot what this was - I've long delete log files I no longer need/want lol so I can't go back and re-read it]. So far the UCC.log file hasn't gotten over 5MB a day yet....so I guess I must have figured out the culprit...?

Oh there *could* be another problem as I discovered this whilst playing on the server that it crashed mid-game and was gonna wonder what that was due by by the server auto reset and wiped the log file clean and restarted from scratch...

Any suggestions or commands I can use to cut back on the amount of noise so that it only logs the parts that you should read? Heh. This'll make the log file even smaller and I can scroll less to get to what I wanna see.



Well there's a script Tripwaire forums that says you can make it restart on crash, but before restarting, it saves the log file into a different directory. I'm guessing you can make a rotation one but of course I wouldn't know how.... I forwarded this to the GSP tech support and they said: "Unfortunately it isn't possible to execute your service via a bat file. This type of script wouldn't work on our TCAdmin panel." And thanks to that I won't be able to diagnose these crashes(or at the one I had) that happen mid-game or on map change....

So what do I do?
 

randomizer

Distinguished
Moderator
Jul 24, 2006
882
0
19,260
62
I know nothing about Killing Floor so I don't know what is amount of logging is normal, but 100GB a day does sound highly unusual. If it is indeed due to a bunch of warnings then you want to find what is causing the warnings, not because the warnings are serious, but because logging them is serious. However, unless you can determine the source of the warnings by reading the warnings you probably won't get any value out of reading the logs. Rather, you'd be better off doing some trial and error until you see a significant reduction in log writes (as it appears you've done, with some success).

I'm not sure exactly how to configure log verbosity so take this with a grain of salt (it is probably better to ask people who are actually familiar with the engine config), but from what I've gleaned you might be able to do so by adding this to your game engine .ini file (back it up first):

INI:
[Core.Log]
global=error
The Core.Log section may already exist and may already contain some entries under it. The Unreal engine allows arbitrary log categories but "global" is a catch-all for anything not explicitly configured. The above will (if it's correct) set the default log verbosity to "error". If the engine follows normal logging conventions this should log anything considered an error or worse. Since you say you've already reduced the log writes you probably don't need to do this though.
 

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0
Do you know anyone here who does so I could PM them or...?



[Core.Log] doesn't exist under KillingFloor.ini anywhere(or User.ini), so I guess I make a new entry and copy and paste what you have there at the bottom of the file? Or is placed somewhere else?

Is there a way to also preserve player join logs so I can also see how populated the server gets from different times and such? If all I see is this and minimized log, that would work out beautifully! :D

No more 100GB logs and maybe I might be able to bring back those leveling perk maps afterwards! :D Not that I'm encouraging power leveling and grinding though.....:D But I don't wanna discriminate just because it's a perk leveling map. All maps are welcome as long as they function as intended and or not crap as I suppose there has to be a line drawn on the quality of maps.....because you know I'm gonna add a map that's not finished(unless the author has given up on it in which case it will never be finished - unless someone else takes it up and finishes it) for example or and has a lot of bugs and glitches that are game breaking, etc.
 

randomizer

Distinguished
Moderator
Jul 24, 2006
882
0
19,260
62

Nope, sorry.


INI files generally aren't ordered so it should not matter where it goes as long as you don't put it in the middle of an existing section. It's probably safest to stick it at the beginning or the end.


You'd need to add another line for that particular category to override the global setting. Probably something like this based on what I can see in the log you provided already:

INI:
[Core.Log]
global=error
PlayerJoinLog=verbose
 

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0
I tried that and it doesn't seem to work, am I suppose to see a completely empty log except for Error: prefixed lines? If so, yeah no it didn't work. I even tried it with just:

INI:
[Core.Log]
global=error
by it self at the top of the ini file and doesn't work, even tried at the bottom and doesn't work....

Code:
Log: Log file open, 10/27/16 18:40:19
Init: Name subsystem initialized
Init: Detected: Microsoft Windows NT 6.1 (Build: 7601)
Init: Version: 3339 (128.29)
Init: RO_VERSION: 1094
Init: KF_VERSION: 1064
Init: Compiled: Dec  2 2014 09:52:28
Init: Command line: KF-ZimsHomeBase_Final_WS.rom?game=KFMod.KFGameType?VACSecured=true?MaxPlayers=6?Mutator=MutLoader.MutLoader?AdminName=Admin?AdminPassword=$B$ -log=.\logs\KFserver.log
Init: Character set: Unicode
Init: Base directory: E:\SteamCMD\KFServer\System\
Init: Ini:KillingFloor.ini   UserIni:User.ini
Init: Build label:  Build UT2004_Build_[2004-11-11_10.48]
Init: Object subsystem initialized
Log: Browse: KF-ZimsHomeBase_Final_WS.rom?Class=Engine.Pawn?Character=Corporal_Lewis?team=1?Sex=M?game=KFMod.KFGameType?VACSecured=true?MaxPlayers=6?Mutator=MutLoader.MutLoader?AdminName=Admin?ADMINPASSWORD=XXXXX?Name=â°â°_ðð¬ð©ð±ð±ð¢
Log: Collecting garbage
Log: Purging garbage
Log: Garbage: objects: 55062->55061; refs: 649017
Log: Game class is 'KFGameType'
Log: STEAMAUTH : Steam auth system activation successful
Log: Bringing Level KF-ZimsHomeBase_Final_WS.myLevel up for play (30) appSeconds: 6.884000...
Log: (Karma): Autodetecting CPU for SSE
Log: (Karma): Using SSE Optimizations
Log: (Karma:) KAggregateGeomInstance: (StaticMeshActor2) Cannot 3D-Scale Karma primitives (sphere, box, cylinder).
ScriptLog: Loading Admins & Groups
ScriptLog: Kick and Ban Privileges Loaded
ScriptLog: Maps & Game Privileges Loaded
ScriptLog: Admins & Groups Management Loaded
ScriptLog: Extra Privileges Loaded
ScriptLog: User Admin already in user list, please choose another name
ScriptLog: Mutators MutLoader.MutLoader
ScriptLog: mut.added MutVisibleSpecsX.MutVisSpecs
ScriptLog: mut.added MutRETROLAR.MutRETROLAR
ScriptLog: mut.added KFDamagePopup.mutDamagePopup
ScriptLog: mut.added KFARGBuchonOPQ.KFARGBuchonOPQ
ScriptLog: mut.added KFDoorMessagesV3.SnitchFrag
ScriptLog: mut.added KFPatHPLeft.MutPatHPLeft
ScriptLog: mut.added KFDoorMessagesV3.KFDoorMessagesV3
ScriptLog: mut.added KFCtryTags.CtryTags
ScriptLog: mut.added KFARGChat.KFARGChat
ScriptLog: mut.added AdminPlus_v3_English.MutAdminPlus
ScriptLog: GameInfo::InitGame : bEnableStatLogging False
Warning: Missing ClassProperty ClassProperty KFMod.KFLevelRules.ItemForSale
ScriptLog: KFLRules = KF-ZimsHomeBase_Final_WS.KFLevelRules
ScriptLog: Game length = 2
PlayerJoinLog: Starting PlayerJoinLog version 101
ChatFilter: Loading Chat Filter
Log: Opening user log ..\UserLogs\ChatFilter 27-10-2016 (18 40 25).log
Log: Defaulting to false
Log: Defaulting to false
WebAdmin: Loading Available Maps
WebAdmin: Loading Game Types
WebAdmin: Unknown Mutator in use:  AdminPlus_v3_English.MutAdminPlus
WebAdmin: XWebAdmin.UTServerAdmin Initialized on port 8075
ScriptLog: [~] ServerAdsKF WebAdmin loaded
Log: Resolving tripwireinteractive.com...
ScriptLog: [~] Starting ServerAdsKF version: 101
ScriptLog: [~] DeeZNutZ - deeznutz@san.rr.com
ScriptLog: [~] BadStreak - [url=http://www.badstreak.com]http://www.badstreak.com[/url]
ScriptLog: [~] There are 5 lines in the list
Log: STEAMAUTH : Sending updated server details - Western Australia |Whitelisted/Greylisted maps and modes| The Friendly Fire (TFF) 0 | 6
Log: STEAMAUTH : Connected to Steam Servers
Log: STEAMAUTH : VAC Security Enabled
Log: STEAMSTATS: SECURITY CHECK FAILED - ../System/ServerColorMut.u
Log: STEAMSTATS: SECURITY CHECK FAILED - ../System/MutLoader.u
Log: STEAMSTATS: SECURITY CHECK FAILED - ../System/AdminPlus_v3_English.u
Log: STEAMSTATS: SECURITY CHECK FAILED - /KFCHARPUPPETS.U
Log: STEAMSTATS: SECURITY CHECK FAILED - ../System/KFCharPuppets.u
Log: STEAMSTATS: Stats Are Now Enabled
Log: Resolved tripwireinteractive.com (198.23.194.228)
Log: STEAMAUTH : Sending updated server details - Western Australia |Whitelisted/Greylisted maps and modes| The Friendly Fire (TFF) 0 | 6
Log: STEAMAUTH : Sending updated server details - Western Australia |Whitelisted/Greylisted maps and modes| The Friendly Fire (TFF) 0 | 6
NetComeGo: Open myLevel 10/27/16 18:40:42 192.168.1.250:61201
NetComeGo: Close TcpipConnection 192.168.1.250:61201 10/27/16 18:40:42
NetComeGo: Open myLevel 10/27/16 18:40:42 192.168.1.250:61201
NetComeGo: Open myLevel 10/27/16 18:40:42 192.168.1.250:61202
Log: STEAMSTATS: SECURITY CHECK FAILED - ../System/ServerColorMut.u
Log: STEAMSTATS: SECURITY CHECK FAILED - ../System/MutLoader.u
Log: STEAMSTATS: SECURITY CHECK FAILED - ../System/AdminPlus_v3_English.u
Log: STEAMSTATS: SECURITY CHECK FAILED - /KFCHARPUPPETS.U
Log: STEAMSTATS: SECURITY CHECK FAILED - ../System/KFCharPuppets.u
. . .
 

randomizer

Distinguished
Moderator
Jul 24, 2006
882
0
19,260
62
Well I did say I wasn't sure exactly how to do it. I could not find any clear official documentation on how to configure logging in the INI file so I had to piece bits of information together. It does seem to be loading the correct file. Maybe this is a question better suited to the Tripwire forum. They might have a better solution than simply adjusting the logging anyway.
 

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0


Well if a rotation script is available, though you will need to know how to script it of course - I don't so I can't do it - otherwise I could check it out for home use but obviously won't be able to implement that through TCPAdmin, that is another option, so is the script I linked earlier about backing up the main log file after a crash; however this script cannot be implemented into TCPAdmin unfortunately, even if they were to do it from their end as I was told. But if you're hosting the server from home, yeah this'll work but I'm not, nor want to as that would mean I need a computer that is always on 24/7 that is also connected to the net and have a *stable* net connection that never ever drops out if ever - which only Fiber users get and I'm on copper......? I have no such net connection... :(

I suppose I could buy out a piece of a data center place and set it up so that it is similar to a home setup that doesn't use GSP's TCPAdmin panel thingy and use that as I own the little bit of space they have there...? They have stable net connection that never ever drops out, so think T5/T9 equivalent lines, over here they are apparently referred to as E rather than a T... Not sure how much it would cost to buy out a piece from a Data-center building near me. Though that doesn't sound very economical at all if all I'm gonna host is this one server at a data center that I bought out..... I suppose I could use their superior internet download and upload speeds for personal things like uploading videos to youtube with glorious >1GB/s upload speed or download a movie with >1GB/s download speed.
 

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0


Sorry for late reply, I must have already read the email notification but forgot about it until now as I was wondering what I should do with the 200GB file.....heh

I'm not sure if it's possible or if there's even a command for it as this(shown below) doesn't work....

[Core.Log]
global=error
PlayerJoinLog=verbose

 

newbie12

Distinguished
Mar 24, 2011
25
0
18,580
0


Well all that is in UCC.log file, and that's the main log file that gets created and over written in the case of resets/restarts. If you're suggesting it also creates a seperate log file for this specific "PlayJoinLog" logs, then I don't believe so.....
 

Ralston18

Dignified
Moderator
Fair enough - I really do not have any direct way to know either way.

I just tend to see if more information can be found. I.e., via Reddit I found this: "Logs are stored under <KillingFloor2ServerDirectory>/KFGame/Logs/".

Read that as a sign that there are other accessible logs. And "verbose" also got my attention as that indicates some level of detail may be available.

Again, no way for me to directly check.