Sorry for late reply, been busy....
Ralston18 :
But your requirement is just to have one continuous "scroll" as I understand your post.
Yeah something like that an if it could read straight of the FTP server, that would be nice as well as that would save me from downloading the file onto my computer, which requires me waiting for it to be downloaded(luckily I'm on unlimited data so downloading 200GB is no problem(though on slow ass ADSL2+ 1MB down it is for me as I don't like waiting and leaving my computer on overnight for a few days for this to happen
), but that might change in future when I move houses, etc.) and *then* read it from there.
Ralston18 :
My first thought is that some demarcation ("chunks of 10 MB") is necessary simply because your computer, storage, software, website, etc. may not be able to handle "single viewing of such a large file.
ok fair enough. But what about in chunks of 100MB, or even 1GB? Surely it can read these length logs, can't it? I know locally, if yo u want to view a relatively large log/txt file, you need to open it in wordpad.........as notepad apparently can't open these large files....
Ralston18 :
If the data is a log file you might be able to download the log file into a database with each log entry being one text record in the database. Or some field structurer like Time, Date, Entry....
Which would require me to download the file and spend the time to sort this out...?
Ralston18 :
That said can you provide more details about the data itself?
Just a log of a game server I have up and running, or did anyways....apparently the problem my Game Service Provider (GSP) has said is that they will kick me off if I ever use up more than hundreds of GB for a log file.....mind you, I only had two main log files that was still there - the 200GB was which I was gonna delete after confirming I didn't download a corrupt file and the actual log file that was created whilst the server is still running - I'm guessing it must have got pretty big because they've told me I have 700+GB worth of what appears to be log files and that it was damaging their hardware blah bla bla....
I went to go ahead and download the whole 200GB file(this was maybe a few or so days before this happened), took a while so I could scope out what could be bloating the file to the point of crashing the GSP or at least causing problems for them for them to threaten to kick me off their list if it happens again. Apparently it's happened several times but I didn't know that.....and I'm guessing this could be the causes of these crashes, but GSP has it setup so that there's no downtime and resets server if it ever crashes, the problem with this is that it over writes the same log file so I am unable to retrieve and view it before deleting it to see what the problem is....
Anyways, so I go to login to check for myself if it's true or not and to maybe get that 700GB file or read somehow.....hahaha.....but they already locked me out so I couldn't do anything...so after a while, we agree on me fixing it up and that they'll unsuspend my account/service so that I can do it. ...and of course they reinstalled the service so lost some of the mods and stuff, they could have warned me....but ehh.....but at least I have a backup of these on my computer to re-upload so it wasn't that much of a loss....
So yeah, here I am trying to find a solution...and thought maybe the solution was to read the huge log file....for any errors....unless you guys have a better and quicker idea, in which case I'm all ears....?
Ralston18 :
Then you would have the added ability to search and sort - depending on the log file format.
Well I can already search, I only need to read, not write anything further to it. So Notepad or a log/txt viewer is enough to search for anything and let me view the lines for any errors.... Just that noneof the readers I have can open it, and the only reader tat opens it, can only read in a maximum of 10MB chunks and since this is 200GB file, I would have to flip through 200k pages.....so me thinking a one time scroll through the entire thing with crtl+f searching for anything of relevance and skipping the bloat lines......could work......might take a full day to go through that I guess......but if it means finding any errors and causes to these silent crashes that I mostly don't notice because I'm not always monitoring my server 24/7 as it runs.....then I was gonna give it a shot.
randomizer :
What do you really want to do with this log file? I can't believe you actually want to read 200GB of text.
I guess read it...? hahaha
I don't know, got a better idea of tracking down errors than go through a 200GB log file specifically looking for error like lines...?
Actually the initial idea of reading the log was to see how popular my server was in the public and that's it. Now that my GSP is pretty much forcing and bugging me to fix this or else they'll shut it down for good, I got back to this again and yeah....
Anyways, that's what logs are for right, reading, ye?